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Throughout  Alberta,  school  systems  are  increasingly  gathering  feedback  and 
perspectives  from  beneficiaries  and  stakeholders  of  the  education  system.  By 
increasing  communication  and  listening  to  ideas  and  opinions,  school  systems 
who  conduct  survey  research  are  gaining  greater  insight  into  the  ideas, 
attitudes,  and  opinions  of  the  Albertans  they  serve.  By  conducting  satisfaction 
research,  education  providers  are  becoming  better  prepared  to  address  the 
needs  and  expectations  of  Alberta  students,  parents  and  teachers. 

This  reference  guide  has  been  designed  to  assist  education  providers  to  plan, 
design  and  implement  satisfaction  surveys.  There  are  a  variety  of  methods 
and  techniques  that  can  be  employed  to  conduct  surveys.  All  have  merits  and 
drawbacks  to  consider  when  deciding  how  to  implement  a  survey.  The 
information  presented  in  this  reference  guide  is  intended  to  help  those  who 
are  involved  in  satisfaction  surveys  within  Alberta's  education  system. 


School  jurisdictions  and  schools  within  Alberta  are  required  to  develop  education  plans  that 
incorporate  performance  outcomes  and  measures.  Often,  these  outcomes  and  measures  are 
related  to  the  satisfaction  of  parents,  students,  staff  and  other  community  members.  The 
information  presented  in  this  reference  guide  can  be  used  to  help  school  jurisdictions  and 
schools  plan  and  implement  surveys  to  measure  satisfaction  within  these  constituent  groups. 

It  is  worth  noting  that  Alberta  Education  conducts  annual  satisfaction  surveys  with  students, 
parents  and  the  public.  The  questions  asked  in  these  surveys  can  be  found  on  the  Internet  at 

http://www.education.gov.ab.ca/ educationsystem/ planning.asp. 

Several  school  authority  representatives  (1 0)  in  Alberta  participated  in  a  survey  that  provided 
valuable  information  for  this  reference  guide.  Their  input  and  suggestions  has  been 
incorporated  in  this  guide  and  practical  examples  are  highlighted.  HarGroup  Management 
Consultants,  Inc.  is  grateful  to  these  representatives  and  acknowledges  their  constructive  and 
valuable  suggestions. 


Inquires  regarding  conducting  satisfaction  surveys  should  be  directed  to: 
Alberta  Education 

Performance  Measurement  and  Reporting  Branch 

9th  Floor,  Commerce  Place 

10155-  102  Street 

Edmonton,  Alberta,  T5J  4L5 

E-mail:  measurement,learning@gov.ab.ca 
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Why  Measure  Satisfaction 

Satisfaction  research  can  enrich  and  enhance  decision  making  processes  by  providing  various 
stakeholders  (students,  parents,  teachers  and  citizens)  an  opportunity  to  provide  their 
perspectives  on  the  quality  of  Alberta's  education  system.  By  understanding  what 
stakeholders  think  about  the  system  and  how  it  serves  the  community,  education  providers 
can  make  improvements  to  more  effectively  address  their  priorities,  values  and  ideas. 

Education  providers  typically  have  many  opportunities  to  gather  feedback  from  stakeholders 
of  the  education  system.  Jurisdiction  or  school  representatives  can  speak  directiy  to  students 
in  school  hallways  or  at  student  union  meetings.  Parents  can  provide  feedback  at 
parent/ teacher  meetings  or  other  organized  events.  Teachers  can  express  their  opinions  at 
curriculum  meetings  or  face-to-face  discussions  with  administrators.  Citizens  can  attend 
school  board  meetings  and  present  opinions  about  how  the  education  system  is  perceived  in 
the  community.  All  of  these  feedback  mechanisms  provide  opportunities  for  stakeholders  to 
express  their  views  and  opinions,  and  help  jurisdictions  and  schools  determine  priorities  to 
meet  the  expectations  of  the  community.  A  satisfaction  survey  is  another  tool  that  enables 
jurisdictions  and  schools  to  understand  stakeholder  perspectives.  The  advantage  of  a  survey 
is  that  it  enables  measurement  of  the  perspectives  of  all  who  are  being  served  by  the 
education  system. 

Most  school  boards  and  schools  within  Alberta's  education  system  already  use  satisfaction 
surveys  to  address  and  understand  a  variety  of  critical  issues.  For  example,  satisfaction 
surveys  are  used  to  plan  education  programs,  examine  use  of  technology  in  schools,  and 
measure  students'  safety  and  security  within  and  outside  of  school.  Satisfaction  surveys  are 
also  conducted  by  school  boards  and  schools  to  address  the  specifications  of  the  Alberta 
Government  Accountability  Framework,  which  is  an  ongoing  process  that  enables  school 
boards  to  implement  continuous  improvement  initiatives  (Alberta  Education  publishes  a 
Guide  for  School  Board  Planning  and  Results  Reporting). 

Satisfaction  surveys  also  help  school  boards  and  schools  to  track  and  analyze  stakeholder 
feedback  over  time,  thus  enabling  them  to  identify  and  understand  changes  in  stakeholder 
perspectives.  Some  school  boards  and  schools  conduct  annual  satisfaction  surveys  with 
various  stakeholder  groups.  Others  survey  target  populations  at  regular  intervals  (e.g 
students  from  Grades  4,  8  and  12  are  surveyed  every  three  years).  Whichever  approach  is 
used,  school  boards  and  schools  can  analyze  the  data  over  time  to  identify  changes  in 
perceptions,  priorities  and  expectations. 

Who  Should  i3e  Surveyed 

School  boards  and  schools  in  Alberta  conduct  satisfaction  surveys  with  a  variety  of 
populations,  including  the  direct  beneficiaries  of  the  education  system  such  as  students  and 
parents  or  guardians  of  students.  Other  beneficiaries  can  be  included  in  satisfaction  surveys. 
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such  as  employers  within  the  school  jurisdiction  and  the  general  citizenry.  These  latter 
stakeholder  groups  receive  indirect  economic  and  social  benefits  from  the  education  system 
such  as  a  skilled  workforce,  citizenry  prepared  to  contribute  to  society,  etc. 

Service  providers  may  also  be  targeted  in  satisfaction  measurement  processes.  Teachers, 
administrators,  and  school  board  members  are  often  surveyed  about  job  satisfaction, 
education  delivery,  infrastructure  priorities,  use  of  technology,  etc. 

Conducting  Satisfaction  Surveys  -  5  Phases 

The  survey  process  is  commonly  conducted  in  five  key  phases,  which  are  broadly  presented 
in  the  diagram  on  the  next  page.  All  phases  are  important  to  the  overall  success  of  the 
satisfaction  measurement  process.  Each  survey  project  undertaken  in  Alberta's  education 
system  will  be  distinct  and  may  include  all  or  some  of  the  issues  presented  in  the  diagram, 
but  most  will  follow  this  framework. 

The  first  two  phases  involve  planning  the  survey  process  and  designing  the  tools  and 
instruments  that  will  be  used  in  the  survey.  These  phases  of  the  process  direcdy  impact  the 
remaining  phases  and  should  be  given  careful  consideration.  For  example,  if  a  survey 
objective  is  not  identified  in  the  first  phase  (and  subsequently  questions  are  not  designed  to 
address  the  issue  in  the  second  phase),  it  may  be  impossible  to  address  the  issue  once  the 
third  phase  (administration  of  the  survey)  is  complete. 

The  third  and  fourth  phases  encompass  conducting  the  survey  with  stakeholders,  analyzing 
the  results,  and  communicating  the  results  to  interested  individuals  or  groups.  The  logistical 
steps  in  conducting  the  survey  will  depend  on  the  method  of  data  collection  selected  for  the 
survey.  For  example,  in  a  web-based  survey,  respondents  may  be  invited  to  participate  in  the 
survey  by  receiving  an  e-mail  invitation  or  receiving  a  letter  that  invites  them  to  participate. 
Conversely,  in  a  telephone  survey  interviewers  contact  respondents  and  verbally  invite  them 
to  participate  in  the  survey. 

Survey  data  will  also  need  to  be  organized  and  assembled  for  analysis  such  as  recoding  and 
collapsing  data  into  categories,  preparing  data  tables,  and  using  subgroup  analysis  to 
determine  whether  different  respondent  groups  have  similar  or  distinct  opinions  about 
issues.  The  survey  findings  should  be  organized  and  presented  in  a  report  that  interested 
individuals  or  groups  can  use. 

In  the  final  phase,  education  providers  are  encouraged  to  develop  and  implement  initiatives, 
or  adjust  existing  policies  and  programs,  based  on  the  survey  results.  In  some  cases,  the 
survey  results  may  suggest  that  the  school  board  or  school  is  addressing  the  expectations  and 
priorities  of  stakeholders.  Alternatively,  the  results  may  identify  specific  issues  that  need  to 
be  addressed  by  the  school  board  and  the  school. 

As  a  final  task,  jurisdictions  and  schools  should  consider  the  successes  and  challenges 
associated  with  the  survey  process  and  identify  ways  that  may  improve  future  surveys. 
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Figure  1   -  Phases  of  a  Satisfaction  Survey 
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A  Checklist  for  Survey  Projects... 

The  following  checklist  provides  a  broad  overview  of  elements  that  should  be  considered 
before,  during  and  at  the  end  of  survey  research  projects.  Each  project  will  be  distinct  and 
may  include  all  or  some  of  the  issues  presented  in  the  checklist. 

□   Has  the  business  need  for  the  survey  research  been  established  and 
Cetting  articulated  (e.g.  how  does  the  survey  relate  to  an  education  plan)? 

started  ^         ^  Survey  Plan  been  developed  (survey  description,  objectives,  target 

populations,  methodology,  use  of  external  contractors,  timelines, 
deliverables,  budget  requirements,  communications,  reporting,  and  use  of  results)? 
n   Does  the  survey  project  have  appropriate  approvals  (e.g.  School  Board,  Superintendents, 
etc)? 

□  Has  a  Request  for  Proposal  been  developed  within  established  guidelines  and  with 
relevant  project  information  and  specifications  (to  distribute  to  external  contractors)? 

□  How  will  external  contractors  be  selected  (open  competition,  single  source,  or  selected 
competition)? 

□  Have  proposals  from  external  contractors  been  rigorously  evaluated  and  approved? 

□  Has  a  formal  contract  been  signed  by  external  contractors  and  jurisdiction  or  school 

representatives? 

□   Are  data  collection  methods  appropriate  for  the  target  populations 
During  ^^^d  project  circumstances? 

implementation...  ^   Have  staff  who  deal  with  the  public  been  informed  about  the  survey 

so  that  they  can  answer  telephone  inquiries  about  it? 
n   Has  the  sample  plan  been  given  adequate  consideration? 

□  Have  the  survey  instruments  been  designed  effectively  and  field  tested? 

□  Have  quality  control  measures  been  established  and  employed  in  the  survey 
administration? 

□  Are  the  specifications  of  the  Freedom  of  Information  and  Protection  of  Privacy  Act 
being  maintained? 

□  Have  the  survey  data  been  adequately  analyzed? 

n   Does  the  survey  report  identify  the  business  need,  describe  the  methodology,  provide  a 
respondent  profile  and  present  factual  findings  obtained  in  the 
research? 

Project  ^   Has  the  project  manager  adequately  evaluated  the  survey  research 

project,  identifying  strengths  and  weaknesses  and  suggesting 
recommendations  for  future  research? 
Have  the  deliverables  been  approved  by  the  person/ s  responsible  for  the  survey  or 
senior  administration? 

Have  the  results  of  the  research  been  communicated  to  the  appropriate  stakeholders  and 
constituents  (both  internal  and  external)? 

Have  the  survey  results  (raw  data  and  reports)  been  sent  or  submitted  for  data  storage  or 
external  distribution? 

Have  the  survey  report  and  data  been  appropriately  warehoused  and  stored  within 
Alberta  Education  protocols  and  Freedom  of  Information  and  Protection  of  Privacy  Act 
specifications? 
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Planning  a  Satisfaction  Survey 

Every  survey  project  should  have  a  plan  that  presents  key  information  about  the  project.  The 
plan  will  help  those  engaged  in  survey  research  to  organize  the  tasks  required  to  successfully 
design  and  execute  the  satisfaction  measurement  process.  It  will  also  help  to  inform  others 
within  the  school  board  or  school  about  the  satisfaction  survey,  its  purpose,  the  process  and 
how  the  results  might  contribute  to  the  overall  decision  making  process  or  education  plan  of 
the  school  board  or  school.  As  well,  a  documented  survey  plan  can  facilitate  continuity  and 
clarity  if  the  survey  is  conducted  on  an  annual  basis. 

The  survey  plan  should  show  a  clear  relationship  between  the  survey  results  and  the 
decisions  that  will  be  made,  the  information  need,  the  methodology  to  be  implemented,  and 
the  reporting. 

Develop  a  description  of  the  satisfaction  survey  project 

When  preparing  a  description  of  the  satisfaction  survey,  those  involved  in  the  survey  should 
organize  basic  ideas  and  concepts  for  what  the  survey  is  about,  who  is  involved  and  when  it 
will  be  conducted.  A  description  of  a  satisfaction  survey  will  typically  include  the  following: 

The  description  should portray  the  intent,  context,  ^   Identify  the  sponsor  of  the  survey  (for  whom 

and  scope  of  the  satisfaction  survey.  the  survey  is  being  conducted  -  e.g.  school 

board,  school,  etc.). 
□   Identify  the  information  need  (why  is  the 
survey  being  done  and  how  the  results  will  support  planning/decision  cycles). 

□  Determine  the  issues  that  need  to  be  resolved. 

n   Set  priorities  for  issues  that  will  be  addressed  in  the  survey. 

□  Identify  an  individual  within  the  organization  as  the  primary  contact  for  the  survey.  This 
individual  should  be  accessible  to  respondents  who  may  have  questions  about  the 
satisfaction  survey. 

n   Determine  the  project  deliverables  -  data  tables,  written  reports,  presentation  formats 
(web-based,  paper,  or  electronic  files),  etc. 

□  Determine  whether  outside  assistance  is  needed  to  support  the  survey  process 
(consultants,  research  firms,  technology  contractors,  printing  companies,  etc.).  See 
Appendix  A  for  information  about  contracting  outside  consultants. 

Define  Survey  Population 

The  survey  population  represents  the  complete  group  of  c  i    i    ^i   -^i   ■   ah  ^         ^  -j  w 

/^^  r  .  ^  .  ^     \  School  authonties  in  Alberta  use  outside  assistance 

persons  (can  also  be  ob  ects,  businesses,  units,  etc.)  to  ,  /       •,/   n    ^  ^   r.i  ^   ■  ^ 

^,  .  ,    ,  ^  ,      .„  ,  ,-1  to  help  mth  all  or  parts  of  the  survey  project. 

which  the  survey  results  will  be  generalized. 

1.    Identify  target  populations  for  the  research.  In  the  context  of  the  education  system, 
these  may  include: 

•  Students 
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•  Parents 

•  Teachers 

•  Administrative  staff 

•  Custodial  staff 

•  School  Board  Members 

•  Superintendents 

•  Stakeholders 

•  Citizens 

•  Employers 


Ten  Alberta  school  authorities  were  asked  which 
grade  levels  they  survey  in  their  satisfaction  projects. 
Most  surveyed  three  grade  levels  (e.g.  grades  4,  8 
and  12)  in  any  given  jean  Few  school  boards 
survey  students  in  ECS  or  grades  1,  2,  and  3. 


The  overall  satisfaction  survey  project  may 
involve  all  or  some  of  the  target  populations 
identified  above. 


•  Determine  or  estimate  the  size  of  the  target 
populations.  The  size  of  the  population  will 
be  needed  if  a  sample  of  the  population  is 
contacted  to  participate  in  the  survey. 

Identify  factors  or  characteristics  of  the  target  population  that  might  limit  the  ability  to 
contact  members  of  the  population  (e.g.  lack  of  direct  contact  information  for  parents, 
vacation  schedules,  access  to  computers,  etc.),  and  address  them  to  minimize  bias  in  the 
survey  results. 

Identify  key  research  questions  or  objectives 

Establish  a  set  of  key  research  questions  or  survey  objectives  to  specify  the  purposes  for 
which  information  is  required. 

•  Identify  the  issues,  problems  and 
hypotheses  to  be  tested  in  the  satisfaction 
survey. 

•  Ensure  that  potential  users  (decision 
makers,  staff,  etc.)  can  understand  what 


Survey  objectives  are  statements  that  clearly  describe 
whatjou  want  to  learn  or  understand  from  the 
survey  results. 


issues  the  survey  research  addresses. 

Develop  survey  objectives  in  partaership  with  decision  makers  and  stakeholders. 
Ensure  that  objectives  are  clear  and  concise. 

The  final  results  of  the  satisfaction  survey  should  address  the  survey  objectives  that  are 
established. 
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Establish  Project  Budget  and  Timelines 

Determine  the  timelines  for  the  project  including  milestones  for  key  phases  and  the 
completion  deadline.  The  following  is  a  simple  illustration  of  a  survey  project  schedule 
showing  tasks  and  timelines.  A  survey  project  may  require  more  details  to  identify  specific 
tasks  and  responsibilities  for  various  individuals  or  staff  members  involved. 


Survey  Project  Schedule: 

Week 

Task: 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

Prepare  survey  objectives  and  budgets  ^| 

Design  survey  methodology  and  sample 

plan  HI 

Design  survey  instruments 

Pre-test  survey  instruments 

Prepare  and  send  out  survey  questionnaires 

Respondents  complete  and  return 

- 

B 

questionnaires 

Clean  and  analyze  data 

Prepare  reports 

Communicate  results  to  relevant 

stakeholders 

Develop  action  plans  to  address  survey 
results 

Key  Milestones:  ♦t* 

In  many  cases,  the  survey  project  may  not  be  conducted  in  sequential  or  linear  steps.  Survey 
administrators  should  anticipate  when  various  activities  might  be  undertaken  within  a  survey 
project  and  plan  appropriate  completion  dates  for  each  activity.  For  example,  programming 
of  scanning  technology  might  occur  during  the  data  collection  phase  so  that  completed 
questionnaires  can  be  scanned  immediately  following  the  data  collection  deadline. 

XX^en  estimating  survey  project  costs,  schools  and  school  boards  should  consider  both  out- 
of-pocket  financial  costs  and  non-dollar  or  indirect  costs.  Some  of  the  survey  costs  that 
estimators  might  need  to  consider  include: 
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Cost-savers: 

•  Use  local  specialists  for  data  collection 

•  Assign  clerical  staff  to  perform  non-technical, 
routine  tasks 

•  Refer  to  existing  survey  instruments 

•  Borrow  whenever possible  (equipment,  staff, 
materials  and  sipplies). 


•  Internal  staff  -  the  time  and  wages  of 
internal  staff  to  organi2e  and  implement  the 
survey  project. 

•  Travel  -  internal  staff  may  be  required  to 
travel  for  the  survey  project  (e.g.  visiting 
schools,  attending  meetings,  etc.). 

•  Printing  -  preparation  of  paper 
questionnaires,  reports,  and  other  related 
materials. 


•  Communications  -  may  include  printing,  postage,  telephone  rentals  and  long-distance, 
couriers,  presentations,  etc. 

•  Data  capture  and  processing  -  costs  may  be  incurred  from  purchasing  speciali2ed 
computer  software  (or  contracting  data  capture  and  processing  to  outside  organizations). 

•  Supplies  and  equipment  -  basic  supplies  such  as  pencils,  pens,  paper,  and  envelopes  may 
be  needed.  Special  equipment  such  as  computers,  projectors,  easels,  etc.  may  be  needed 
to  implement  the  survey  and  present  the  results. 

•  Outside  consultants  -  all  or  part  of  a  satisfaction  survey  project  may  be  contracted  to 
outside  consultants.  Consultants  commonly  charge  rates  on  a  per  diem  basis  or  as  a  fixed 
rate  for  completing  a  contracted  service.  The  next  section  of  this  manual  examines  issues 
related  to  engaging  outside  consultants. 


Contracting  Outside  Contractors/Consultants 

Outside  research  contractors/ consultants  are  sometimes  engaged  to  assist  in  designing  and 
executing  a  survey  project,  or  conducting  a  specific  component  of  the  project. 

There  are  typically  three  methods  of  sourcing  and  selecting  outside  consultants: 

1.  Open  Competition  by  Advertisement  -  Service  requirements  are  advertised  to  the 
general  public  and  all  interested  parties  are  invited  to  submit  a  proposal  (e.g. 
advertisements  in  newspapers).  This  method  can  often  result  in  a  large  number  of 
proposals  being  submitted  for  review. 

2.  Single  Source  Selection  —  Only  one  contractor/ consultant  is  invited  to  submit  a 
proposal  for  the  provision  of  services.  Generally,  organizations  should  have  a 
compelling  reason  for  using  this  selection  process  (e.g.  only  available  supplier,  limited 
budget,  etc.).  It  is  important  to  ensure  that  the  contractor/ consultant  has  appropriate 
qualifications  and  experience  to  conduct  the  project,  if  this  method  is  chosen. 

3.  Select  Competition  by  Resource  List  -  Specified  contractors/consultants  are 
iQvited  to  respond  to  a  Request  for  Proposal.  At  least  3  to  5  contractors/consultants 
should  be  invited  to  respond.  It  should  be  noted  that  some  contractors  may  not 
submit  proposals  for  projects;  therefore  organizations  may  want  to  invite  more  than  3 
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contractors  to  obtain  at  least  3  proposals  for  consideration.  A  resource  list  can  be 
developed  by  posting  a  Request  for  Qualifications  on  the  Alberta  Purchasing 
Connection  (http:/ / w^ww.purchasingconnection.ca/).  Contractors  will  then  submit 
their  qualifications  and  can  be  placed  on  a  resource  list. 

Service  contracts  must  comply  with  the  Agreement  on  Internal  Trade  (AIT).  For  more 
information  about  AIT,  see  http:/ /www.iir.gov.ab.ca/trade  policy /interprovincial.asp. 

i.  Request  for  Proposals 

Solicitation  for  services  can  be  conducted  through  a  Request  for  Proposal  (RFP)  process.  A 
RFP  is  a  solicitation  method  used  to  seek  survey  project  management  plans  and  proposed 
costs  from  external  contractors. 

Develop  Request  For  Proposals 

A  Request  for  Proposals  (RFP)  is  a  document  used  to  notify  potential  contractors  that  a 
project  is  about  to  be  initiated,  and  the  organization  is  interested  in  receiving  proposals,  or 
bids,  from  interested  external  contractors.  The  RFP  usually  provides  requirements  and 
specifications  about  the  survey  project  and  requirements  of  the  external  contractors  in 
conducting  services. 

The  RFP  provides  potential  contractors  with  information  needed  to  prepare  and  submit  a 
proposal  to  provide  research  services.  Information  contained  in  the  RFP  will  generally 
include  contact  information,  specifications  regarding  the  proposals  submitted,  and  some 
background  information  regarding  the  project.   Contractors  will  review  the  RFP  to 
determine  whether  or  not  they  feel  they  can  take  on  the  project,  and  whether  or  not  to 
submit  a  proposal  to  offer  services. 

Proposals  can  be  sought  by  posting  a  Request  for  Proposals  on  the  Alberta  Purchasing 
Connection  (http: /  / www.purchasingconnection.ca /) . 
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A  Request  for  Proposal  (RFP)  should  include: 

□  Project  background,  information  and  description, 
n  Survey  objectives. 

□  Use  of  survey  results. 

□  Technical  specifications  of  the  survey. 

□  Timing  of  the  project. 

□  Project  deliverables  and  ownership  of  results. 

□  Proposal  requirements. 

n  Proposal  evaluation,  contract  negotiations  and  award  specifications. 

n  Freedom  of  Information  and  Protection  of  Privacy  Act  specifications  (see  Section  4.3). 

□  Response  instructions. 

n  Period  of  proposal  commitment  (e.g.  30  days  following  the  submission  date). 

The  RFP  may  include  attachments  such  as  detailed  survey  plans,  previous  years' 
questionnaires/ reports,  draft  survey  instruments,  resources  available  for  the  project  and 
other  materials. 

ii.  Proposal(s)  Evaluation  and  Contract 

Development 

Organizations  and/ or  satisfaction  survey  committees  should  develop  factors  to  rate 
proposals,  assign  weights  to  criteria,  and  finalize  evaluation  forms.  In  the  case  of  a 
satisfaction  survey  committee,  members  usually  review  the  proposals  individually  and  then 
meet  to  discuss  and  arrive  at  a  consensus  rating  for  each  proposal. 

Evaluate  Proposals  and  Select  External  Contractor 

There  are  a  number  of  issues  to  consider  when  assessing  the  competence  and  suitability  of 
an  outside  contractor/ consultant  for  a  specific  satisfaction  survey  project.  The  following  list 
provides  a  basic  checklist  for  selecting  an  outside  contractor: 

n   Does  the  proposal  exhibit  an  understanding  of  the  project  and  its  requirements? 

□  Is  the  proposed  methodology  sound? 

□  Does  the  proposal  provide  a  high  probability  of  accomplishing  the  study  objectives? 

□  Is  the  accuracy  of  the  data  ensured? 

□  Is  the  reliability  of  the  data  ensured? 

□  Is  the  approach/ methodology  based  on  sound  research  design  principles? 

□  Is  there  enough  flexibility  to  ensure  success  if  project  parameters  change? 

□  Are  project  timelines  and  costs  reasonable  and  within  specifications? 

n   Does  the  contractor  have  the  technical  expertise  and  experience  to  fulfill  the  project? 

□  Does  the  contractor  have  the  resources  to  complete  the  project? 

Other  criteria  may  also  be  needed  depending  on  specifications  of  the  project. 
A  template  and  example  of  an  evaluation  form  are  presented  in  Appendix  A. 
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The  successful  external  contractor  should  be  informed  of  the  outcome  of  the  selection 
process  and  offered  a  contract  for  service.  Unsuccessful  contractors  should  be  notified  that 
their  proposals  were  not  selected  for  the  project. 


Designing  the  Survey 

There  is  no  one  best  method  to  use  when  conducting  satisfaction  surveys.  What  may  work 
best  for  one  school  board  or  school  may  not  be  the  best  approach  for  another.  Selecting 
methods  to  conduct  a  satisfaction  survey  will  depend  on  several  factors  such  as  the  t}^pe  of 
information  that  is  needed  from  the  survey,  the  targeted  population,  ease  of  contacting 
respondents,  and  the  financial  and  resource  costs  to  carry  out  a  survey. 

This  section  of  the  manual  provides  organizations  with  guidelines  and  specifications  to 
consider  when  deciding  which  survey  design  methods  to  employ. 

Determine  data  collection  technique(s) 

In  choosing  an  optimal  data  collection  method  for  the  survey,  the  nature  and  requirements 
of  the  information  should  be  considered.  The  survey  population  will  also  influence  the  data 
collection  method  chosen  (how  can  the  population  be  contacted  and  surveyed).  It  is 
important  to  consider  these  factors  in  detail,  as  the  data  collection  method  can  substantially 
impact  the  quality  of  information  obtained  by  the  survey. 

The  types  of  data  collection  methods  commonly  used  by  school  boards  and  schools  within 
Alberta's  education  system  include  (presented  in  order  of  frequency  of  use  by  school  boards 
and  schools): 

Mail  survey  -  (a  self-administered  survey)  questionnaires  are  sent 
Commonly  used  to  survey:  ^  respondents  through  the  mail  (or  some  other  delivery  method) 

•  Students  asked  to  complete  the  form  and  return  it.  There  are 

'  variations  of  the  mail  survey  approach  that  might  not  use  postal 

•  ^^^^  ^'  mail.  For  example,  teachers  could  provide  students  with  a 

•  Teachers  questionnaire  to  complete  in  class.  The  students  complete  the 

questionnaire  and  return  it  to  the  teacher  in  a  sealed  envelope. 
Another  example  might  involve  students  taking  questionnaires  home  for  their  parents  to 
complete.  After  the  parents  have  completed  the  questionnaire,  they  put  it  in  a  sealed 
envelope  and  their  children  take  it  back  to  the  school  to  give  to  the  teacher  (or  return  to 
some  other  drop  off  system).  In  interviews  with  Alberta  school  authorities,  the  mail  survey 
technique  was  the  most  common  technique  used  to  survey  students,  parents  and  teachers. 
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Web-based  survey  -  (a  self-administered  survey) 
becoming  a  commonly  used  survey  respondents  are  asked  to  interact  with  a  computer  and  enter 

method:  their  responses  in  the  questionnaire  form  by  using  a 

•  Students  keyboard  or  touching  a  computer  screen.  Some  schools 

'  have  used  this  method  when  surveying  students,  teachers 

•  Teachers,  parents.  Schools  have  organized  time  schedules  so  that 

•  Parents  students  can  use  computer  labs  to  access  computers  and 

complete  the  survey.  Teachers  have  been  invited  to 
complete  questionnaires  at  a  computer  terminal  available  in  the  school.  Some  schools  have 
sent  out  questionnaires  to  parents  with  students,  but  offered  parents  the  option  to  complete 
the  questionnaire  on-line  (in  most  cases,  this  has  had  limited  success).  Some  organizations 
are  employing  innovative  approaches  to  encourage  parents  to  participate  in  satisfaction 
surveys  using  web-based  questionnaires.  For  instance,  parents  are  encouraged  to  visit  a 
school's  computer  lab  prior  to  the  annual  parent/ teacher  interviews  to  complete  a 
questionnaire  on-line. 

Telephone  survey  -  (an  interviewer  administered 
Might  be  used  to  survej:  survey)  trained  interviewers  contact  respondents 

•  Parents  telephone  and  administer  the  interview. 

Interviewers  ask  respondents  questions  and 

•  ^rip  qyers,  record  responses  on  paper  or  computer-aided 

•  Citi^ns  telephone  interview  forms.  The  telephone  survey 

has  been  used  to  conduct  satisfaction  surveys 
with  parents,  local  employers  and  the  public.  Typically,  this  survey  technique  has  not  been 
employed  with  students  or  teachers  because  these  groups  are  mostiy  captive  audiences 
within  the  education  system  and  can  be  efficientiy  interviewed  through  the  mail  or  web- 
based  survey  techniques  (as  presented  above). 

In-person  survey  -  (an  interviewer  administered 
Might  be  used  to  survey:  survey)  trained  interviewers  administer  the 

•  Parents  interview  in-person  at  respondents'  homes, 

offices,  schools,  etc.  The  interviewer  is 

•  mp  je  ,  responsible  for  asking  questions  and  recording 

•  Younger  students  responses  on  paper  or  computer-aided  interview 

forms.  This  survey  technique  is  not  commonly 
used  to  conduct  satisfaction  surveys  within  Alberta's  education  system  but  it  may  present 
some  advantages  when  attempting  to  administer  complex  questionnaires  to  respondents  or 
interviewing  younger  students  for  a  satisfaction  survey  (e.g.  grades  1  to  3) 

In  some  satisfaction  survey  projects,  combinations  of  the  data  collection  methods  presented 
above  have  been  employed  (e.g.  a  mail  survey  conducted  with  parents  and  web-based 
surveys  with  students  and  teachers). 

The  table  on  the  following  page  presents  data  collection  methods  that  are  commonly  used  to 
conduct  satisfaction  surveys  in  Alberta's  education  system,  along  with  considerations  for 
applying  each  technique. 
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Another  consideration  for  deciding  upon  a  data  collection  method  is  the  survey  sampling 
method  and  sample  size.  The  next  section  of  this  manual  examines  this  component  of 
survey  design. 
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Considerations  for  i 

Survey 

Technique 

\pplying  Survey 
Process 

Techniques 

Advantages 

Disadvantages 

Costs 

Response  Rates 

Mail  Survey 

(Self- Administered) 

A  questionnaire 
is  sent  to 
respondents  to 
complete  and 
return  through 
mail. 

Complete  at  leisure 
Detailed  response 
Interviewer  bias  removed 
Can  use  visual  stimuli 
Potential  for  lengthy 
questionnaires 
Respondent  perception  of 
anonymity  is  moderate 

Lack  of  control  over 

verbatim  responses 

Challenges  with  sequencing 

Slow  turn-around 

In  some  cases,  difficult  to 

encourage  response  (e.g. 

parents) 

Low  to  moderate- 
Students  -  moderate 
to  high 

Parents  -  low  to 
moderate 

Teachers  -  moderate 
to  high 

Web-based  Surveys 
(Self-Administered) 

A  surveying 
method  in  which 
the  Internet  (or 
an  intranet)  is 
used  to  collect 
data. 

Questionnaires 
are  distributed  to 
respondents 
using  electronic 
mail,  or 

respondents  are 
asked  to  visit  a 
website  to 
complete  an  on- 
screen 

questionnaire. 

Complete  at  leisure 
Detailed  response 
Interviewer  bias  removed 
Can  use  visual  stimuli 
Quick  turn-around 
Inexpensive  to  add  sample 
Verbatim  responses  tend  to  be 
more  detailed 

Ability  for  complex  skips  and 
rotations 

Potential  challenges  with 
representative  samples 
(respondents  must  have 
access  to  Internet) 
Lack  of  control  over 
verbatim  responses 
Challenges  with  sequencing 
Lower  tolerance  for 
questionnaire  length 
Need  for  de-bugging 
Potential  for  crashes 
In  some  cases,  difficult  to 
encourage  response  (e.g. 
parents) 

Respondent  perception  of 
anonymity  may  be  low 

Low 

Students  -  moderate 
to  high 
Parents  -  low 
Teachers  -moderate 
to  high 

Telephone  Survey 

(Interviewer 

Administered) 

These  surveys 
involve  one-on- 
one  questioning 
and  answering 
between  an 
interviewer  and 
respondent  by 
means  of  a 
telephone. 

Control  over  sequencing 
Relatively  fast-turn  around 
Able  to  control  samples  and 
quotas 

Can  access  widely  dispersed 
samples 

Use  of  computer  aided 
telephone  interviewing  (CATI) 
reduces  costs  with  automatic 
data  entr)' 

Typically,  interviewers  can 
encourage  participation 
Respondent  perception  of 
anonymit}^  is  moderate 

Potential  for  interviewer 
influence 

There  are  limits  to  what 

respondents  can  recall 

under  pressure 

May  be  less  convenient  for 

respondents 

Does  not  permit  visual 

stimuli 

Need  for  CATI  debugging 

Moderate 

Typically  high  rates 
of  return 

In  Person  Survey 

(Interviewer 

Administered) 

A  face-to-face 
meeting  between 
an  interviewer 
and  respondent, 
which  may  be 
conducted  in  an 
office,  home, 
school,  etc. 

Control  over  sequencing 
Relatively  fast-mrn  around 
Potentially  able  to  conduct 
longer  interviews 
Potential  for  lengthy 
questionnaires 
Typically,  interviewers  can 
encourage  participation 
Respondent  perception  of 
anonymity^  is  moderate 

Pressure  on  respondent  can 
influence  ability  to  recall 
May  be  less  convenient  for 
respondents 

Reduces  ability  to  supervise 
Greatest  potential  for 
interviewer  influence 
Can  be  affected  by  weather 

High 

Typically  high  rates 
of  return 

14 


A  Practical  Guide  to  Conducting  Surveys  witliin  Alberta 's  K-12  Education  System 


Determine  sample  method  and  size 

Satisfaction  surveys  may  be  conducted  on  an  entire  population  (i.e.  a  census  survey),  or  on  a 
sample  of  a  population.  This  approach  might  be  used  when  the  survey  population  is  small, 
or  when  comprehensive  data  is  required.  Another  reason  for  conducting  a  census  is  to  allow 
everyone  within  a  population  the  opportunity  to  provide  feedback.  Often  though, 
satisfaction  surveys  are  conducted  on  a  portion  of  the  population  (i.e.  a  sample).  This 
approach  reduces  costs  where  the  population  to  be  surveyed  is  large.  To  provide  reliable 
data  that  can  be  used  to  make  inferences  about  the  population  as  a  whole,  it  is  important 
that  the  sample  survey  be  conducted  on  a  portion  or  sample  of  the  population  that  is 
representative  of  the  whole. 

With  a  census  survey  approach,  all  members  of 
the  targeted  population  are  offered  an 
opportunity  to  participate  in  the  satisfaction 
survey.  This  does  not  mean  that  all  members  will 
participate  (unless  legislated  by  law).  Rather,  each 
member  has  an  opportunity  respond  to  the 
survey,  but  it  is  their  choice  as  to  whether  they 
will  participate  (at  least  in  the  context  of 
conducting  surveys  within  Alberta's  education 
system).  In  most  cases,  the  data  gathered  using  a 
census  approach  will  result  in  a  sample  of  the 
population  (rather  than  a  true  census  where  100% 
of  the  population  is  surveyed).  Where 
substantially  less  than  100%  of  the  population 
responds  to  a  census  survey,  those  responsible 
for  the  survey  analysis  should  check  for  characteristics  of  the  non-respondents  that  might 
bias  the  survey  results. 

Various  sampling  methods  can  be  employed  for  surveys  (see  following  page);  however, 
probabilit}^  samples  should  be  used  whenever  possible.  With  probability  sampling,  each 
member  within  a  population  (or  sub-populations)  has  a  known  probability  (e.g.  an  equal 
opportunity)  of  being  selected  to  respond  to  the  survey  and  inferences  to  the  population  can 
usually  be  drawn  about  the  population  from  the  responses  of  the  sample.  Non-probability 
samples  may  be  biased  because  of  the  way  respondents  are  selected  to  participate  in  the 
survey.  If  a  sample  is  biased,  it  can  be  difficult  (if  not  impossible)  to  draw  inferences  about 
the  entire  population  from  the  data  that  are  gathered. 

Sample  size  is  also  an  important  factor  to  consider  in  the  sample  design.   Essentially,  the 
larger  the  sample,  the  more  certainty  the  survey  researchers  can  have  that  the  data  gathered 
in  the  survey  wiU  be  representative  of  the  opinions  of  the  population.  The  smaller  the 
sample,  the  more  likely  the  data  from  the  sample  will  differ  significantly  from  the  opinions 
of  the  population. 

The  relationship  between  sample  size  and  accuracy  of  findings  is  referred  to  as  sampling 
error  (or  margin  of  error)  and  is  a  measure,  usually  estimated,  of  the  extent  to  which  data 


A  small  survey  of  ten  Alberta  School  authorities 
revealed  that  most  adopted  census  approaches  for 
conducting  satisfaction  surveys.  For  example,  all 
students  in  grades  4,  8  and  12,  and  their  parents, 
were  provided  an  opportunity  to  participate  in  the 
sati  faction  survey.  As  well,  all  teachers  and 
administrative  staff  within  the  authorities  could 
participate  in  the  survey. 

The  census  approach,  however,  might  be  less 
practical  when  surveying  local  employers  or  the 
general  citi^nry. 
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from  the  survey  sample  will  represent  the  entire  population.  Media  often  report  survey 
results  and  present  the  sampling  error  for  the  survey.  For  example,  77%  of  respondents 
were  satisfied,  plus  or  minus  5%.  The  ±5%  represents  the  estimated  sampling  error  for  the 
survey  data.  In  other  words,  the  actual  number  of  satisfied  respondents  likely  lies  within  the 
range  of  72%  and  82%. 
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Sampling  Approaches  and  Methods 
Sampling  Approaches 

Sampling  Methods 

Probability  Samples  -  All  members  of  a 
population  have  a  known  chance  of  being 
selected  into  the  sample 

Systematic  Sampling  -  Starting  at  a  random 
point  within  a  list  of  population  members,  a 
constant  skip  interval  is  used  to  select  every 
other  sample  member. 

Simple  Random  Sampling  -  A  table  of 
random  numbers,  random  digit  dialing,  or 
some  other  random  selection  procedure  is 
used  to  ensure  that  each  member  of  a 
population  has  the  same  chance  of  being 
selected  into  the  sample. 

Cluster  Sampling  -  The  population  is 
divided  into  geographic  areas,  each  of  which 
must  be  very  similar  to  the  others.  A  few 
areas  can  be  selected  to  conduct  a  census  or 
a  sample  can  be  drawn  from  a  select  group 
of  areas. 

Stratified  Sampling  -  If  a  population  is 
expected  to  have  a  skewed  distribution  for 
one  or  more  distinguishing  factors, 
subpopulations  or  strata  can  be  identified 
and  a  simple  random  sample  can  be  drawn 
from  each  stratum.  Weighting  procedures 
may  be  required. 

Non-Probability  Samples  —  The  chance 
that  members  of  a  population  have  been 
selected  into  the  sample  is  unknown. 

Convenience  Sampling  -  A  "high-traffic" 
area  is  used  to  select  respondents  for  a 
sample  (e.g.  a  school  of  high  enrolment,  an 
industrial  area  of  a  community,  etc.) 

Judgment  Sampling  -  A  researcher  uses 
his/her  own  judgment  to  identify  which 
respondents  will  be  in  the  sample. 

Referral  Sampling  -  Respondents  are  asked 
for  names  of  others  like  themselves  who 
might  participate  in  a  survey  (also  known  as 
snowball  sampling). 

Quota  Sampling  -  Quotas  are  identified  for 
sub-populations  of  a  population. 
Respondents  are  screened  to  determine 
whether  they  represent  the  parameters  of  the 
quota. 
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Sampling  error  is  estimated  within  a  confidence  interval.  A  confidence  interval  indicates  the 
probability  (e.g.  95  times  in  100)  that  the  true  value  lies  within  a  specific  range.'  It  is 
common  for  survey  findings  to  rely  on  a  95%  confidence  level,  but  higher  (99%)  and  lower 
levels  (90%)  are  acceptable.  The  level  of  confidence  selected  depends  on  the  degree  of 
reliability  that  is  deemed  acceptable  for  the  survey  findings.  Another  way  of  communicating 
confidence  intervals  is  to  say  "19  dmes  out  of  20,"  which  represents  a  95%  confidence  level. 

Another  point  to  consider  when  determining  sample  size  is  the  types  of  analyses  that  will  be 
undertaken.  For  example,  if  the  researcher  wants  to  compare  results  among  various  schools 
within  an  authority,  consideration  should  be  given  to  increasing  the  overall  sample  size  to 
facilitate  greater  confidence  in  findings  from  the  comparative  analysis. 

The  following  table  provides  sample  sizes  for  various  sampling  errors  at  the  95%  confidence 
level.  For  instance,  if  a  school  has  1 ,000  students,  approximately  286  would  need  to 
participate  in  the  satisfaction  survey  for  the  data  to  achieve  an  estimated  ±5%  sampling  error 
(margin  of  error)  within  a  95%  confidence  level. 


Sample  Size  Table- at  95%  Confidence 
(assuming  a  probability  sample) 

Interval 

Population  Size 

Sample  Size  Based  on  Estimated  Sampling  Error 
±3%               ±5%              ±7%  ±10% 

25 

24 

24 

22 

20 

50 

48 

44 

40 

33 

100 

92 

80 

67 

50 

200 

169 

133 

101 

67 

300 

236 

171 

121 

75 

400 

294 

200 

135 

80 

500 

345 

222 

145 

83 

600 

390 

240 

152 

86 

800 

465 

267 

163 

89 

1,000 

526 

286 

169 

91 

2,500 

769 

345 

189 

96 

5,000 

909 

370 

196 

98 

10,000 

1,000 

385 

200 

99 

25,000 

1,064 

394 

202 

100 

100,000 

1,099 

398 

204 

100 

500,00  or  more 

1,110 

400 

204 

100 

Adapted  from  a  sample  size  table  presented  in  How  Big  Should  the  Sample  Be? 
Statistics  Canada,  1993 

Non-response  is  another  issue  to  consider  when  determining  sample  size.  In  some  cases, 
non-response  may  be  high  (e.g.  a  mail  or  web-based  survey  of  parents)  or  low  (a  web-based 


Source:  How  to  Conduct  Customer  Surveys,  Institute  of  Citizen-Centred  Service,  2001. 
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survey  in  which  students  complete  the  questionnaire  in  a  class  lab).  This  factor  needs  to  be 
considered  when  determining  a  sample  size  for  a  survey. 

Many  school  authorities  in  Alberta  have  adopted  Sample  Frame  -  Any  Hst,  database,  reference 

the  census  approach  to  survey  parents  in  mail  material,  or  device  that  identifies,  and  allows 

surveys.  One  of  the  reasons  for  ijnpkmenting  this  access  to  members  of  a  target  population  could 

approach  is  to  conipensate for  high  non-response  be  used  as  a  sample  frame  for  a  satisfaction 

rates  (between  50%  to  85%)  typically  experienced  survey.  Examples  of  a  sample  frame  within 
in  mail  surveys  of  population.  Alberta's  education  system  might  be  a  list  of 

parents  (with  addresses,  telephone  numbers  or  e- 
mail  addresses),  a  database  of  students'  e-mail 
addresses,  and  telephone  yellow  pages  that  list  companies  in  a  local  area. 

The  sample  frame  used  for  a  survey  should  be  as  comprehensive  as  possible.  Any  member 
information  missing  from  a  sampling  frame  will  cause  sampling  errors.  Researchers  may 
need  to  update  or  improve  the  sample  frame  prior  to  a  survey  to  minimize  the  potential  for 
errors  in  the  sampling  process. 

In  some  cases  (e.g.  a  telephone  survey),  a  sample  frame  will  need  to  have  significantiy  higher 
numbers  of  contacts  in  order  to  obtain  the  desired  sample  size.  For  example,  a  telephone 
survey  may  require  a  listing  of  contacts  that  is  3  or  4  times  higher  than  the  targeted  sample  to 
achieve  the  volume  of  completed  responses  needed. 
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Design  survey  tools  and  instruments  (e.g.  questionnaire) 

A  questionnaire  is  a  tool  comprised  of  a  group  or  sequence  of  questions  that  are  designed  to 
elicit  information  about  one  or  more  subjects  from  a  respondent.  A  well-designed 
questionnaire  should  collect  information  from  a  respondent  efficiendy  and  effecdvely, 
whether  it  is  interviewer-administered  or  self-administered  (i.e.  respondent  completed).  It 
should  be  focused  to  achieve  the  survey  objectives  set  out  in  the  survey  plan,  and  present 
questions  that  are  easily  understood  and  can  be  accurately  answered  by  respondents. 


Several  drafts  of  a  questionnaire  may  he  necessary 
before  the  instrument  is  in  a  form  that  mil 
effectively  address  the  survey  objectives  and  can  be 
administered  efficiently  and  effectively  with 
respondents. 


Typically,  the  questionnaire  is  one  of  several 
instruments  that  are  used  in  a  survey.  Survey 
instruments  involve  introduction  letters,  data 
collection  forms,  reminder  notices,  return- 
envelopes,  computer  software  programs,  etc. 
These  instruments  serve  numerous  functions  in 
conducting  surveys,  for  instance: 


•  Introduce  the  survey  to  the  respondents, 

•  Establish  rapport  with  respondents, 

•  Encourage  survey  participation, 

•  Screen  respondents  to  systematically  identify  appropriate  subjects, 

•  Gather  information,  and 

•  Enable  capture  or  collection  of  the  data. 

The  composition  of  these  instruments  will  have  a  major  impact  on  data  quality,  respondent 
relations  and  interviewer  performance  (where  appropriate).   The  following  descriptions 
present  the  composition  and  organization  of  the  instruments  and  some  key  elements  of 
questionnaire  development. 


Instrument  Composition  and  Organization 

Introduction  -  The  introduction  describes  the  purpose  of  the  survey,  establishes  a  rapport 
with  respondents,  encourages  participation,  and  screens  respondents  for  qualifications. 
Ensure  that  the  value  of  providing  information  is  made  clear  to  respondents  by  explaining 
how  the  information  wiU  be  used.  Respondents  should  also  be  assured  that  their  responses 
are  confidential  and  that  their  participation  is  voluntary.  The  following  elements  should  be 
present  in  the  introduction  section  of  questionnaires: 

•  Identification  of  survey  organization/ sponsor. 

•  Purpose  of  the  survey. 

•  Explanation  of  respondent  selection. 
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•  Request  for  participation/ provide  incentive. 

•  Screening  of  respondent. 

Identification  of  Survey  Sponsor  -  In  the  context  of  conducting  satisfaction  surveys  within 
Alberta's  education  system,  the  sponsor  (school  authority  or  school)  should  be  identified  in 
the  introduction  to  the  questionnaire.  A  contact  person  should  also  be  available  to 
respondents. 

Screening  process  -  A  process  that  systematically  selects  respondents  to  interview  and 
excludes  those  who  do  not  qualify.  The  process  can  also  randomize  the  selection  of 
respondents  (e.g.  "May  I  speak  to  the  person  in  your  household  whose  birthday  comes 
next?"). 

•  Use  screening  questions  that  exclude  the  largest  proportions  of  unqualified  respondents 
first. 

•  Avoid  placing  sensitive  screening  questions  first. 

•  Use  screening  process  to  fill  quotas. 

•  Keep  track  of  call  record  for  estimates  or  weighting. 

•  Use  clear  quota/ termination  instructions. 


Several  schools  have  used  symbols  as  response 
categories  for jounger  students  (grades  1,2,  (&  3) 
to  respond  to  surveys. 


Instructions  -  In  some  cases,  respondents  wiU  be 
presented  with  instructions  to  participate  in  a 
survey  or  complete  a  questionnaire.  Instructions 
may  be  presented  in  a  separate  document  to  the 
data  collection  form  (questionnaire),  at  the 
beginning  of  the  data  collection  form  or  for 
individual  questions  presented  in  the  data 
collection  form.  These  instructions  should: 


•  Be  as  concise  as  possible. 

•  Avoid  emotionally  loaded  terms. 

•  Be  clear,  uncluttered,  and  appealing. 

•  Possibly  use  symbols  or  illustrations  (e.g.  Place  a  check  in  the  appropriate  box  ...  e.g.0). 

Data  collection  forms  -  Data  collection  forms  present  questions  that  address  the  survey 
objectives  and  question  responses  used  to  record  answers  on  the  form.  These  forms  may  be 
printed  on  paper,  or  presented  electronically.  Another  component  of  data  collection  forms 
is  the  equipment  or  materials  that  might  be  used  to  capture  the  data  (e.g.  return-envelopes, 
computers,  hand-held  devices,  etc.). 
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Basic  Questionnaire  Development 

Questionnaires  should  be  designed  to  address  the  following: 

•  Present  relevant  questions  that  are  required  to  address  the  survey  objectives.  In  other 
words,  questions  should  focus  on  the  topic  of  the  survey. 

•  Start  with  easy,  unthreatening,  but  pertinent  questions,  to  build  a  rapport  with  the 
respondent.  Demographic  questions  should  only  be  present  at  the  beginning  of  a 
questionnaire  when  they  are  being  used  for  screening  purposes. 

•  Definitions  should  be  clearly  stated  to  respondents. 

•  Be  as  brief  as  possible. 

•  When  switching  topics  within  a  questionnaire,  use  a  transitional  phrase  to  allow 
respondents  to  'switch'  their  thoughts. 

•  Use  filter  questions  to  let  respondents  avoid  sets  of  questions  that  do  not  pertain  to  them. 

•  Personal  information  gathered  within  questionnaires  should  comply  with  specifications 
presented  in  the  Freedom  of  Information  and  Protection  of  Privacy  Act. 

Question  structure,  wording  and  formatting  should  minimize  any  biasing  of  responses.  At 
the  very  least,  questions  should: 

•  Address  a  single  concept  only. 

•  Not  use  double  negatives,  unfamiliar  words,  abbreviations,  acronyms,  trade  jargon, 
colloquialisms,  etc. 

•  Avoid  humor. 

•  Be  time  specific. 

•  Be  as  concise  as  possible. 

•  Avoid  emotionally  loaded  terms. 

•  Be  clear,  uncluttered,  and  appealing. 

•  Not  lead  respondents  to  provide  a  specific  response. 

•  Avoid  implied  alternatives. 

Types  of  Questions 

There  are  three  main  types  of  questions  posed  to  respondents  in  questionnaires: 
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Open-ended  questions  -  enable  respondents  to  answer  in  their  own  words.  An  open 
question  allows  respondents  to  interpret  the  meaning  of  the  question  and  provide  an  answer 
that  addresses  their  interpretation. 

e.g.  What  do  you  like  about  the  school  your  child  attends? 


Open-ended  questions  are  presented  in  questionnaires  for  several  reasons: 

•  To  provide  respondents  an  opportunity  for  self-expression 

•  To  allow  respondents  to  elaborate  on  complex  issues 

•  To  allow  a  full  range  of  responses 

•  To  allow  for  clarification 

•  To  obtain  'natural'  wording 

•  To  gather  numerical  data  (e.g.  In  what  year  were  you  bom?  1 9  ) 

•  To  add  variety  to  the  questionnaire 

There  are  challenges  with  open-ended  questions,  for  both  the  respondent  and  the  researcher 
that  is  conducting  the  satisfaction  survey.  For  respondents,  open-ended  question  can  be 
demanding  and  require  more  thinking  than  closed  ended-questions.  As  well,  it  can  be  more 
time  consuming  for  the  respondent  to  answer  an  open-ended  question.  In  terms  of 
researchers  who  conduct  surveys,  it  can  be  time  consuming  and  difficult  to  record  the 
responses,  there  is  likely  a  need  to  code  the  data  into  categories,  and  it  can  be  difficult  to 
analyze  and  interpret  the  data. 

Closed-ended  Questions  -  respondents  are  offered  prescribed  answers  in  which  they  can 
choose.  Essentially,  respondents  are  restricted  to  choosing  an  answer  or  response  option 
that  is  specified  to  them. 

There  are  generally  three  types  of  closed-ended 
Two-choke  or  multiple-choice  questions  are  used  to         questions  commonly  used  in  satisfaction  surveys 
determine  whether  one  alternative  ivill  be  favored  within  Alberta's  education  system,  including: 

over  the  rest,  or  if proportions  of  the  population 

tend  to  prefer  various  alternatives.  Two-choice  question  (only  two  choices  are 

available  to  the  respondent): 

e.g.       Are  there  any  children  in  your  household  who  are  attending  elementary,  junior  high 
or  senior  high  school  in  Alberta? 

□  Yes   n  No 

Multiple-choice  question  (more  than  two-choices): 
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e.g.       Are  you  currently  enrolled  in  ... 

□  a  public  school  □  a  separate  school  □  a  private  school 

Checklist  question  (check  as  many  choices  as  apply): 

A  checklist  question  is  used  when  researchers  are  „rr,        i  •       i  i  •  i  i  •  /  i 

, .  ,      r  -J  ■  e.ff.       What  sub  ects  have  you  studied  this  term  (select 

interested  in  how  open  a  particular  response  option         all  that  a  M 


is  chosen  by  respondents  or  the  frequeng  by  which 
one  option  is  chosen  over  others 


□  English   □  Math    □  Science   □  Social  Studies 


Closed  ended  questions  are  usually  easy  and  fast  for  respondents  to  answer  and  are  easy  for 
the  researcher  who  conducts  the  survey  to  code  and  analyze.  Closed-ended  questions  can 
take  effort,  on  the  part  of  the  researcher,  to  develop  and  may  oversimplify  an  issue. 


Scale-Response  Questions  -  Response  scale  questions  are  instruments  used  to  measure 
phenomena  (e.g.  issues,  experiences,  perceptions,  etc.)  that  cannot  be  easily  assessed  by 
direct  means  (e.g.  observable  incidents  of  behavior).  Respondents  are  asked  to  consider 
their  response  or  answer  to  a  question  based  on  a  scale,  range  or  rank. 


There  are  numerous  types  of  scaled  response 
Scaled  questions  are  used  to  determine  a  level  of  questions.  The  following  provides  some 

measurement  about  an  issue,  topic  orsubjea.  examples  of  scale-response  questions  that  might 

Ranking  questions  are  typically  used  to  identify  be  used  in  a  satisfaction  survey  about  the 

respondents' favorability  among  different  options.  education  system: 


Do  you  agree  or  disagree  that  your  school  provides  a  safe  environment? 

□  Agree   □  Disagree 

Overall,  how  satisfied  or  dissatisfied  are  you  with  the  quality  of  education  you  receive  in  your 
school?  Are  you... 

□  Very  satisfied   □  Satisfied   □  Dissatisfied   □  Very  Satisfied 

Please  rank  each  subject  in  terms  of  your  preference.  Place  a  '1'  by  your  first  choice,  a  '2'  by 
your  second  choice,  and  so  on. 

 English   Math   Science   Social  Studies 


24 


A  Practical  Guide  to  Conducting  Surveys  witliin  Alberta 's  K-12  Education  System 


The  Performance  Measurement  and  Reporting 
Branch  of  Alberta  Education  uses  a  4  point 
satisfaction  scale  for  its  annual parent,  student, 
and  teacher  survey: 


There  is  no  industry  standard  for  the  type  of  scale 
questions  that  might  be  used  in  each  and  every 
situation.  Any  number  of  question  items  and 
scale  categories  may  be  created  depending  on  the 
nature  of  the  issues  that  are  being  investigated  in 
the  survey  research. 


e.g.         Very  satisfied  Overall,  when  selecting  a  response  scale  question, 

Satisfied  researchers  should  consider  whether  the  question 

Dissatisfied  (and  its  response  scale)  is: 

Very  dissatisfied. 

•  Easy  for  respondents  to  understand. 

•  Easy  for  researchers  to  interpret. 

•  Minimizing  response  bias. 

•  Easy  to  distinguish  between  point  intervals. 

•  Relevant  to  the  business  decision. 


Additional  issues  to  consider  when  developing  or  using  response  scales  questions  include: 

Regardless  of  the  number  of  categories  selected  in  the  response  scale,  researchers  should  not 
compare  or  perform  comparative  analysis  of  dissimilar  and  unrelated  scales  (because  interval 
relationships  among  different  scales  cannot  be  correlated).  For  example,  the  results  of  a  4- 
point  scale  should  not  be  compared  to  those  of  a  5-point  scale.  As  well,  a  scale  using 
'satisfaction'  categories  should  not  be  compared  with  'agreement'  categories. 

Researchers  should  not  report  neutral  or  middle  category  responses  to  odd  numbered  scales 
as  affirmative  or  negative  opinions  or  perceptions,  unless  an  unbalanced  scale  has  been 
specifically  articulated  to  respondents  (limitations  to  unbalanced  scales  are  presented  later  in 
this  section). 


Do  Not  Report  Neutral  or 
Affirmative  or  Negative  0{ 
How  satisfied  or 
dissatisfied  are  you,  using 
a  scale  of: 

Very  satisfied 
Satisfied 

Neither  satisfied  nor 
dissatisfied 
Dissatisfied 
Very  Dissatisfied 


liddle  Category  as 
nion  or  Perception 
How  satisfied  or 
dissatisfied  are  you, 
using  a  scale  of: 

1  being  very  satisfied, 
and 

5  being  very  Dissatisfied 


Middle  Category'  can  be 
Reported  As  Affirmative 
How  satisfied  or  dissatisfied 
are  you,  using  a  scale  of: 

Ver)^  strongly  satisfied 
Strongly  satisfied 
Somewhat  satisfied 
Somewhat  dissatisfied 
Dissatisfied 
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Researchers  should  enquire  with  other  researchers  and  Alberta  Education  as  to  response 
scale  questions  being  used  in  similar  satisfaction  surveys  to  enable  comparative  analysis 
across  studies.  Typically,  Alberta  Education  researchers  have  used  balanced  (see  below)  or 
equally  weighted  affirmative  or  negative  4  or  5  point  scales  in  survey  research. 


The  following  examples  depict  scales  commonly  used  in  satisfaction  surveys: 

Satisfaction  scale  -  How  satisfied  or  dissatisfied  are  you  with  ...?  Please  use  a  scale  of  Very 
satisfied',  'satisfied',  'dissatisfied',  and  'very  dissatisfied.'  (add  'neither  satisfied  nor  dissatisfied' 
added  as  a  mid-point  for  a  5  point  scale). 

Performance  scale  -  How  would  you  rate  ...?  Would  that  be  'very  good',  'good',  'fair'  and 
'poor'  (add  'excellent'  as  a  starting  point  for  a  5  point  scale)? 

Expectation  scale  -  Compared  to  what  you  expected,  how  would  you  rate  ...?  Would  you 
say  'much  better  than  expected',  'better  than  expected',  'about  as  expected',  'worse  than 
expected'  and  'much  worse  than  expected'. 

Priority  scale  -  How  important  is  ...  to  you?  Please  use  a  scale  of 'not  at  all  important',  not 
important',  'important'  and  'very  important'. 

Improvement  scale  -  Indicate  the  amount  of  improvement,  if  any,  that  is  needed?  Would 
you  say  'none',  'slight',  'some',  'much'  or  'huge'. 

Alberta  Education  conducts  annual  satisfaction  surveys  with  students,  parents  and  teachers 
in  the  province.  Schools  and  school  boards  involved  in  survey  research  can  review  the 
questionnaires  used  in  these  surveys  by  accessing  the  following  web  pages: 

http:/ /www.education.gov.ab.ca/ pubstats /research. asp 

Also,  Appendix  B  presents  a  four  step  process  for  researchers  to  use  when  developing  scale 
response  questions. 


Balanced  4-Point  Scale 
Very  sadsfied 
Satisfied 
Dissatisfied 
Very  dissatisfied 


Balanced  5-Point  Scale 
Yery  satisfied 
Satisfied 

Neither  satisfied  nor  dissatisfied 

Dissatisfied 

Very  Dissatisfied 
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Pre-test  survey  tools  and  instruments 

Survey  tools  and  instruments  (including  questionnaires)  should  be  tested,  under  field 
conditions,  prior  to  implementation.  Essentially,  pre-testing  is  an  informal  test  of  the  tools 
and  instruments  usually  administered  in  the  same  manner  as  planned  for  the  survey.  The 
entire  questionnaire  can  be  tested,  or  a  portion  of  the  questionnaire  may  be  tested  (mainly  in 
cases  where  most  of  the  questions  have  been  asked  in  previous  surveys). 

Pre-tests  are  undertaken  to: 

It  can  he  helpful  to  have  individuals  not  directly 

involved  in  the  survey  project  remeiv  the  •  Discover  vague  question  wording,  or  poor 

questionnaire.  The^  may  find  issues  ordering. 

{conprehension,  logic  of  structure,  clanty  of  ,  Identify   errors   in   questionnaire  layout. 


instruction,  etc.)  that  are  not  readily  apparent  to 
the  researcher  mana^ng  the  satisfaction  survey 

p^oj^f^f^  •  Identify  questions   that  respondents  are 


instruction,  or  sequencing. 

Identify  questions  that 
unable  or  unwilling  to  answer. 


•  Develop  response  categories. 

•  Detect  bias  in  questions. 

•  Determine  suitability^  of  the  questionnaire  for  measuring  concepts. 

•  Measure  interview  length  and  refusal  pattems. 

The  size  of  the  pre-test  sample  should  depend  on  the  specifications  of  the  project  (e.g. 
population  or  survey  sample  size,  questionnaire  previously  used  in  past  surveys,  etc.).  The 
average  pre-test  consists  of  approximately  10-15  completed  interviews. 

Pilot  testing  may  also  be  necessary,  especially  for  large  and  complex  surveys.  Pilot  testing  is 
essentially  a  "dress  rehearsal"  and  duplicates  the  final  survey  design  on  a  small  scale  from 
beginning  to  end,  including  data  processing  and  analysis.  Pilot  testing  provides  an 
opportunity  to  refine  the  questionnaire,  as  well  as  the  overall  survey  administration  process. 

It  may  be  necessary  to  test  the  survey  instruments  more  than  once. 
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Administering  the  Survey 

Prepare  Instruments,  Staff  and  Equipment  for  Survey 
Administration 

Normally,  there  are  a  variety  of  tasks  to  undertake  to  prepare  for  the  data  collection  process. 
Tasks  may  differ  depending  on  the  data  collection  technique  chosen  for  the  survey. 

Mail  surveys 

•  Closed  ended  questions  may  need  to  be  pre- 
Some  school  authorities  have  partnered  or  shared  coded  to  facilitate  data  entr}\ 

resources  with  other  authorities  to  conduct  surveys.  ,  jf  scanning  equipment  is  being  used  for 

coding,  questionnaires  may  need  to  be 
e.g.  sharing  questionnaire  scanning  equipment  formatted     to     accommodate  scanning 

technology. 

•  Survey  instruments  will  need  to  be  printed  (introduction  letters,  questionnaires, 
instruction  sheets,  self-addressed  stamped  envelopes,  etc.) 

•  Questionnaires  may  need  to  be  pre-coded  with  identification  numbers  or  printed  on 
various  colored  paper  to  identify  respondents  or  sub-sets  of  respondents. 

•  Sampling  protocols  should  be  implemented  to  ensure  randomized  samples. 

•  Contact  labels  (for  envelopes)  may  need  to  be  developed  from  the  sample  frame  and  put 
on  envelopes. 

•  Envelopes  will  need  appropriate  postage. 

•  Survey  instruments  may  need  to  be  put  into  envelopes  for  distribution. 

•  An  altemative  to  the  above  tasks,  a  mail  distribution  house  might  be  contracted  to 
organize  and  disseminate  survey  instruments  to  respondents. 
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Telephone  surveys 

•  Questionnaires  may  be  pre-coded  and  printed  (if  data  recording  is  paper-based). 
Alternatively,  questionnaires  may  be  programmed  for  use  in  a  computer-aided  telephone 
interviewing  system  (a  computer  software  program  that  enables  simultaneous 
interviewing  and  data  entr}^). 

•  Call  record  sheets  may  be  printed  using  data  from  the  sample  frame  (see  an  example  in 
Appendix  C).  Another  option  is  to  use  a  call  management  software  program  (which  may 
be  part  of  the  computer-aided  telephone  interviewing  system).  In  either  case,  sampling 
protocols  should  be  implemented  to  ensure  representative  samples  (if  probability  samples 
are  being  used). 

•  Train  telephone  interviewers  for  survey  administration.  A  basic  training  program  might 
include: 

°     Instruction  on  interviewing  protocols 
An  interviewer  training  program  may  take  3  to  6  (techniques,  building  relationships, 

hours  to  adequately  instruct  interviewers  on  the  recording  data,  ethics,  etc.) 

tasks  and  responsibilities.  °     Interviewers  need  to  establish  a 

friendly  relationship  with 
respondents,  gain  cooperation  and 

trust. 

°     Introduction  and  screening  is  very  important  in  building  the  relationship  between 

interviewer  and  respondent. 
°     Interviewers  should  approach  each  interview  as  though  it  were  to  take  place 

immediately. 
°     Answer  respondents'  questions. 

°     Interviewers  should  always  be  pleasant  and  professional  in  interviewing  situation. 
°     Vocal  expressions  should  possess  clear  enunciation,  moderate  rate  of  speech,  low 
pitch  and  inflection. 

Interviewers  should: 

•  Ask  every  question  exactiy  as  worded  and  structured  in  the  questionnaire. 

•  Ask  questions  in  a  positive  manner. 

•  Repeat  and  clarify  questions  that  are  misunderstood  or  misinterpreted. 

•  Probe  for  clarification  of  responses 

•  Listen  to  full  answers  provided  by  respondents. 

•  Interviewers  should  not: 

•  Suggest  answers  to  a  respondent 

•  Ask  leading  questions  when  probing  or  clarifying  responses. 

•  Provide  personal  information. 

•  Offer  personal  opinions  about  survey  issues. 
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•  Brief  interviewers  on  the  background  of  the  project. 

•  Brief  interviewers  on  survey  instruments  and  record  call  sheets 

•  Go  through  each  question  with  interviewers.   Ensure  they  understand  each  question, 
instructions  for  question,  skip  patterns. 

•  Explain  proper  data  recording  procedures. 

Go  through  record  call  sheets  to  ensure  interviewers  are  familiar  with  the  process. 
Have  interviewers  practice  interviewing  (such  as  interviewing  each  other)  with  the  survey 
instruments. 

Web-based  surveys 


^search  reveals  that  respondents  are  more  likelj 
to  participate  in  a  multi-page  web-based  survey 
than  a  single  page  web-based  survey  that  requires 
them  to  scroll  down 


Closed  ended  questions  may  need  to  be  pre- 
coded  to  facilitate  software  programming. 


•  Questionnaires  will  need  to  be  programmed 
into  a  survey  computer  software  program. 
The  software  program  may  be  web-page  or 
e-mail  based,  or  an  electronic  file  attached  to 
an  e-mail  message.    (Note:  this  step  in  the  process  may  require  specialized  computer 
software  or  expertise). 

Sampling  protocols  should  be  implemented  to  ensure  randomized  samples. 
Invitations  will  need  to  be  sent  to  sample  that  is  derived  from  the  sample  program. 


In-person  surveys 

•  Questionnaires  may  need  to  be  pre-coded  and  printed  (if  data  recording  is  paper-based). 
Alternatively,  questionnaires  may  need  to  be  programmed  for  use  on  a  hand  held  device 
or  laptop  computer  for  computer-aided  telephone  interviewing. 

•  Contact  record  sheets  and  protocols  will  need  to  be  organized  to  guide  interviewers  in  the 
selection  of  respondents. 

•  Train  interviewers  for  survey  administration.  The  basic  training  program  is  presented 
above  (telephone  survey). 
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Collect  data  from  respondents 

Survey  instruments  (or  invitations  to  participate)  should  be  sent  to  respondents  in  self- 
administered  surveys.  In  interviewer  assisted  surveys,  interviewers  should  begin  the 
interview  process  (within  a  day  or  two  of  their  training  session). 

During  data  collection,  the  following  points  must  be  considered: 

•  Respondents  should  always  be  given  a  contact,  if  they  request,  in  case  they  have  questions 
or  concerns  they  wish  to  express.  Respondents  should  be  able  to  check,  without 
difficulty,  the  identity  and  validity  of  the  organizations  or  individuals  contacting  them  for 
the  survey  project. 

•  Respondents  should  always  be  informed  that  the  survey  is  voluntary  and  that  they  do  not 
need  to  give  a  reason  for  declining  participation.  They  should  not  be  misled  when  being 
asked  for  their  cooperation. 

•  Respondents  should  be  given  an  accurate  estimate  of  interview  length. 

•  The  survey  needs  to  clearly  state  how  the  results  will  be  used  and  reported. 

•  The  respondents  must  be  told  how  the  issue  of  confidentiality  will  be  handled  and 
whether  or  not  their  answers  can  be  identified  in  the  results. 

In  self-administered  surveys,  respondents  should  be  provided  with  information  that  will 
enable  them  to  complete  the  questionnaire. 

In  interviewer- administered  surveys,  interviewers  should  have  training  to  properly  administer 
questionnaires  to  respondents  through  telephone  interviews. 

Supervisors  should  monitor  flow  rates  (completion  rates  per  hour)  to  control  costs  and 
ensure  interviewers  are  maintaining  efficient  completion  rates. 

Quality  control  measures  should  be  established  and  employed,  such  as: 

•  Appropriate  sample  control  procedures  should  be  organized  and  implemented  for  all  data 
collection  operations.  These  procedures  may  track  the  completion  status  of  surveys, 
monitor  quotas,  establish  flow  rates,  etc. 

•  Effective  control  systems  should  be  established  to  ensure  the  security  of  data  transition 
and  handling. 

•  Interviews  should  be  monitored  or  confirmed  (i.e.  least  10%  of  completed  interviews 
should  be  confirmed  for  telephone  and  in-person  surveys). 

•  100%  data  entry  verification  should  be  conducted  for  collection  methods  that  do  not 
employ  computer-aided  data  entry  (e.g.  computer-aided  telephone  interviewing). 
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Prepare  data  for  analysis 


The  satisfaction  survey  data  will  need  to  be  cleaned  and  edited  in  preparation  for  analysis. 
The  cleaning  and  editing  process  involves  reviewing  questionnaires  to  increase  accuracy  and 
precision.   In  a  review,  researchers  might  identify  illegible,  incomplete,  inconsistent  or 
ambiguous  responses.  As  well,  re-coding  of  data  may  be  necessary  to  further  enhance  the 
use  of  the  data. 


Several  school  authorities  contract  out  the  data  A  codebook  should  be  estabUshed  to  ensure 

capture,  cleaning  and  editing  and  tabulation  internal  consistencies  widiin  the  dataset. 

components  of  the  survey  project.  Codebooks  can  be  very  helpful  if  more  than  one 

individual  will  be  involved  in  the  data  cleaning 
and  editing.  For  instance,  a  codebook  can  be 
used  to  train  data  entry  staff.  Codebooks  contain  instructions  and  necessary  information 
about  the  data  set.  It  is  common  for  every  question  (variable)  to  have  descriptions  of  what 
might  be  contained  in  responses  to  the  question.  For  example,  a  codebook  might  have  the 
following  description  for  a  question 


variable  name 
question  number 
response  label 


Instructions 


satisfaction  with  education 

Question  3 

l=very  satisfied 

2=satisfied 

3=dissatisfied 

4=very  dissatisfied 

5=don't  know 

Recode  label  5  as  missing  for  data 
analysis 


Researchers  should  run  a  set  of  frequencies  to  reveal  the  number  of  responses  for  each 
question  and  response  category.  A  review  of  these  frequencies  will  give  the  researcher  a 
rough  check  for  completeness  and  accuracy  of  the  data  (e.g.  responses  should  not  exceed  the 
total  number  of  respondents,  responses  should  not  show  be  out  of  range  values,  etc.). 

Data  cleaning  involves  checks  of  data  consistency  and  missing  responses.  Consistency 
checks  identify  data  that  are  out  of  range,  logically  inconsistent  or  have  extreme  values. 
These  types  of  errors  are  inadmissible  in  the  data  set  and  must  be  corrected  (i.e.  re- 
contacting  respondents)  or  discarded. 

Re-coding  (variable  transformation)  involves  transforming  data  to  create  new  variables  or 
modify  existing  variables.  This  process  is  common  with  open-ended  questions  where 
responses  are  re-coded  into  consistent  categories. 

Once  a  researcher  is  confident  that  the  data  is  properly  cleaned  and  edited,  the  data  set  is 
ready  for  analysis. 
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Analyze  data 

Data  analysis  is  the  process  of  transforming  raw  data  into  useable  information.  The  basic 
steps  of  the  analytic  process  consist  of  examining  the  issue  and  asking  meaningful  questions, 
and  developing  support  for  answers  that  are  communicated  to  decision  makers  and  other 
readers.  To  be  an  effective  support  for  decision  making,  the  data  must  be  analyzed 
appropriately.  The  analysis  should: 

•  Examine  the  issue  and  ask  meaningful  questions 

•  Generally,  follow  guidelines  for  analysis  accepted  in  the  field  of  survey  research. 

•  Effective  data  analysis  typically  focuses  on  issue,  theme,  and  idea  categorization,  rather 
than  simply  presenting  the  survey  data. 

•  Apply  statistical  significance  tests  to  address  hypotheses,  where  appropriate. 

•  Caution  should  be  observed  in  drawing  conclusions  concerning  causality.  In  the  absence 
of  certainty  that  a  specific  cause  is  the  only  one  consistent  with  facts,  cite  all  possible 
explanations,  not  just  one. 

•  Develop  support  for  answers 

•  Use  frequencies,  cross  tabulations  and  statistical  tests  to  identify  proportional 
representations  (percentages)  and  correlations  among  data. 

•  Indicate  the  rationale  for  the  selection  of  any  significance  tests  used. 

•  Indicate  details  about  any  transformations  (use  of  Z-Scores,  for  example)  made  on  the 
data. 

•  Outline  any  conclusions  drawn  from  the  analysis  or  limitations  of  the  analysis. 

Various  types  of  statistical  analyses  may  be  applied  to  the  data  to  reveal  issues,  trends,  associations, 
etc. 

Descriptions  of  central  tendencies  reveal  typical,  average  or  representative  values  of  the  data  set  - 
mean  (average),  median  (middle  value  -  half  the  data  are  larger,  the  other  half  smaller),  and  mode 
(most  frequentiy  occurring  value). 

Other  descriptive  statistics  such  as  frequencies,  percentiles  and  percentages  summarize  the 
data  by  revealing  distributions  of  responses.  In  satisfaction  surveys,  it  is  common  to  identify 
the  percentage  of  respondents  who  express  satisfaction  (e.g.  very  satisfied  or  satisfied). 
Cross-tabulations  examine  the  relationships  among  two  or  more  variables.  For  example, 
different  schools  against  satisfaction  levels). 

Statistical  significance  tests  reveal  notable  differences  within  the  data  set  (Chi-square,  z 
scores,  t-tests,  analysis  of  variance,  etc.) 
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Multivariate  analysis  is  conducted  to  determine  relationships  and  associations  between  and 
among  various  variables  in  the  data  set  (e.g.  factor  analysis,  regression  analysis,  etc.) 

An  analytical  tool  commonly  used  in  satisfaction  research  is  an  importance/ satisfaction 
matrix.  This  analysis  involves  plotting  ratings  from  satisfaction  and  importance  scaled- 
response  questions  on  a  grid  and  helps  define  priorities  for  service  delivery.  The  grid 
enables  researchers  to  visually  identify  areas  for  service  improvement. 


Service  Improvement  Priority  Matrix 


HIGH 


•a 

u 


Quadrant  1 

Concentrate  Here 

-  Important  to  stakeholders 

-  Poor  performance 

-  Expectations  not  being  met 

Quadrant  2 

Keep  Up  the  Good  Work 

-  Important  to  stakeholders 

-  Good  performance 

-  Expectations  being  met 

Quadrant  3 
Low  Priority 

-  Less  important  to  stakeholders 

-  Poor  performance 

-  Not  priorities  for  improvement 

Quadrant  4 

Possible  Overkill 

-  Less  important  to  stakeholders 

-  Good  performance 

-  Exceeding  expectations 

-  May  present  opportunity  to 
re-allocate  resources 

1 

LOW 


HIGH 


Satisfaction  Ratings 
(Average  Response) 


A  school  authority  might  ask  two  scaled-response  questions  with  1 5  items  related  to  delivery 
of  education  services.  The  first  question  asks  respondents  to  answer  using  an  importance 
scale,  the  second  question  a  satisfaction  scale  (Note:  if  an  item  is  asked  in  terms  of 
importance,  it  is  also  asked  in  the  context  of  satisfaction. 

Average  scores  for  importance  and  satisfaction  ratings  plot  the  mid-point  on  each  axis  of  the 
matrix.  The  researcher  then  plots  the  data  in  the  matrix.  Those  items  that  fall  in  the 
Quadrant  1  should  be  given  further  consideration.  Those  that  land  in  Quadrant  3  are 
considered  low  priorit}^ 
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Communicating  Survey  Results 

Identify  stakeholder  groups  that  will  receive  survey 
results 

The  documentation  of  survey  results  serves  as  a  record  of  what  was  done  during  the  survey 
research  to  provide  a  context  for  effective  and  informed  use  of  the  results.  Reporting 
should  provide  a  complete,  unambiguous  and  multi-purpose  record  of  the  survey,  including 
the  data  produced  by  the  survey. 

Knowing  the  intended  audience  for  the  report 
Several  school  authorities  present  survey  results  to  can  influence  the  way  a  satis  faction  survey  report 

Parent  Councils.  is  prepared.  Some  issues  that  researchers  should 

consider  when  preparing  a  report,  include: 

•  What  audiences  will  be  reviewing  and  using  the  report  (administration,  teachers.  Alberta 
Education,  the  general  public,  etc.)? 

•  Will  readers  of  the  report  understand  survey  concepts  and  terminology? 

•  Will  audiences  expect  to  read  interpretation  of  the  data  in  the  report? 

•  What  key  themes  arise  from  the  data  results  (a  good  report  typically  conveys  several  key 
themes  from  the  data)? 


Determine  methods  to  report  survey  results 

Reporting  may  employ  multiple  forms  (e.g.  paper,  electronic,  visual,  etc.)  and  address  the 
needs  of  different  audiences  and  purposes.  It  is  important  to  consider  the  reporting 
expectations  of  interested  individuals  or  groups  when  deciding  which  method  is  appropriate. 

Survey  results  can  be  presented  in  various  forms,  such  as: 

•  Oral  reports  are  often  used  to  inform  interested  individuals  or  groups  of  survey  results. 
Typically,  oral  reports  are  accompanied  by  a  visual  presentation  (using  an  overhead 
projector  or  computer  and  LCD  projector)  that  shows  graphical  illustrations  of  the  survey 
findings. 

•  An  executive  summary'  might  identify  key  points  in  narrative  that  come  out  of  a 
satisfaction  survey  (a  one  or  two  page  briefing). 
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A  data  report  presents  tables  of  response  counts  and  proportions  for  all  questions  asked  in 
the  satisfaction  survey.  There  may  be  litde  narrative  presented  in  the  report,  other  than  an 
introduction  that  provides  the  survey  objectives  and  methodology. 

A  descriptive  or  narrative  report  usually  summarizes  and  explains  the  survey  findings, 
provides  interpretation  and  describes  various  analyzes,  and  identifies  key  themes  arising  from 
the  data. 

Data  from  the  survey  can  also  be  reported  in  a  variety  of  ways,  for  example: 
Tables 


Satisfaction  with  Education 

Response 
Choice 

%  of  Respondents 

All  respondents 
(n=  1,200) 

Grade  4 
Students 
(n=400) 

Grade  8 
Students 
(n=400) 

Grade  11 
Students 
(n=400) 

Very  satisfied 

52 

55 

53 

50 

Satisfied 

32 

29 

32 

35 

Dissatisfied 

10 

11 

9 

8 

Very  Dissatisfied 

6 

5 

6 

7 

Total 

100 

100 

100 

100 

(Note:  the  n=  shows  the  number  of  respondents  in  each  category) 


Charts 


Satisfaction  with  Education 


rim 


Very  satisfied  Satisfied  Dissatisfied        Very  dissatisfied 

%of  Respondents 

m  All  respondents  (n=1 ,200)   ■  Grade  4  Students  (n=400) 
□  Grade  8  Students  (n=400)   □  Grade  1 1  Students  (n=400) 
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Prepare  survey  results  report 

To  enable  a  clearer  understanding  of  the  results,  survey  reporting  should  (particularly  in  a 
descriptive  or  narrative  report): 

•  Identify  the  survey  objectives 

•  Describe  methodologies 

•  A  description  of  the  survey  population  (students,  parents,  teachers,  etc.). 

•  Details  of  the  sampling  method,  size,  and  frame. 

•  When  technically  relevant,  a  statement  of  response  rates  and  a  discussion  of  any  possible 
bias  due  to  non-response. 

•  The  precision  (sampling  error)  and  confidence  levels. 

•  The  timeframe  in  which  the  survey  was  conducted. 

•  The  method  (s)  by  which  the  data  was  collected. 

•  Describe  respondents 

•  A  demographic  profile  of  respondents  should  be  presented  in  the  report.  Comparisons 
of  the  sample  with  the  target  population  might  be  made  to  determine  whether  the  sample 
is  representative. 

•  Present  survey  results  and  the  relevant  factual  findings  obtained. 

•  The  level  of  detail  provided  in  the  report  will  depend  on  the  intended  audience,  the 
medium  of  dissemination,  and  the  intended  use  of  the  survey  data. 

•  Comparison  of  actual  results  to  past  performance  where  possible. 

•  Use  graphs/ figures  in  addition  to  text  or  tables  to  communicate  messages. 

•  In  presenting  rounded  data,  use  the  number  of  significant  digits  that  is  the  maximum 
number  consistent  with  the  utility  of  the  data  (e.g.  4.6%  rather  than  4.57869%). 

•  The  draft  report  should  be  checked  and  any  errors  corrected  (e.g.  consistency  of  figures 
used  in  the  text,  tables  and  charts,  verification  of  accuracy  of  external  data,  and  simple 
arithmetic). 

•  Conclusions  presented  in  the  report  should  be  consistent  with  the  evidence  available  in 
the  survey  data. 
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Other  reporting  issues 

Provide  a  description  of  limitations  in  interpreting  the  results. 

When  aggregating  and  releasing  data  outside  the  organization,  data  cells  with  counts  of  less 
than  six  should  be  suppressed.  Although,  in  some  circumstances,  this  number  may  be  as 
high  as  ten  where  the  information  could  be  used  to  identify  an  individual  respondent  or 
small  group  of  respondents.  Sensitivity  of  data  cell  counts  is  usually  determined  by  the  need 
to  protect  individual  characteristics  of  respondents.  Methods  typically  used  to  transform 
data  to  protect  personal  information  include: 

Collapsing  Categories  -  Data  are  grouped  (or  re-coded)  into  cell  categories  so  that  none  of 
the  cells  are  considered  sensitive. 

Cell  Suppression  -  Sensitive  cells  are  deleted  from  a  table.  It  should  be  noted,  however, 
that  it  may  be  possible  to  obtain  the  value  of  the  suppressed  cell  by  solving  a  system  of  linear 
equations  based  on  other  cells  in  a  table. 

Rounding  -  The  data  cell  value  is  rounded  to  a  number  that  would  reasonably  protect 
personal  information  of  respondents  (e.g.  5  or  10). 

Stripping  -  The  removal  of  any  names  and  other  personal  identifiers  from  records,  while 
leaving  other  information  such  as  opinion  data  in  the  records. 

Organize  and  document  references 

•  Include  copies  of  the  survey  instruments  (and  other  relevant  technical  data)  in  the 
appendices. 

•  The  report  should  be  subject  to  extensive  review  by  researchers,  decision  makers,  and 
other  relevant  staff,  as  appropriate,  to  ensure  quality  and  readability.  Reports  should  be 
edited  meticulously. 

•  A  communications  expert  (in-house  or  agency)  might  be  consulted  to  review  for 
sensitivity  of  issues  and  plain-language  and  to  determine  if  a  plan  or  strategy  is  needed  to 
distribute  results  to  external  audiences. 

•  Presentations  may  be  required  to  School  Board  Members,  Principals,  Stakeholders,  etc. 

Reporting  and  disclosure  of  personal  information  (e.g.  names  of  respondents,  individual 
personal  characteristics,  etc.)  must  comply  with  the  privacy  protection  provisions  of  the 
Freedom  of  Information  and  Protection  of  Privacy  Act  and  the  FOIP  Regulation  (see  Other 
Issues  Related  to  Satisfaction  Surveys). 
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Communicate  results  to  interested  individuals  or  groups 

The  key  messages  communicated  to  interested  individuals  and  groups  should  reflect  the 
primary  findings  of  the  survey  and  address  the  information  needs  of  applicable  audiences. 

Those  who  have  been  engaged  in  survey  research  are  encouraged  to  share  survey  results  with 
internal  staff,  administrators  and  boards,  and  external  stakeholders  and  interested  parties. 

In  some  cases,  data  may  be  sent  to  Alberta  Education  as  required  in  the  Accountability 
Framework  of  the  department. 
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Implementing  Satisfaction  Survey 
Results 

Develop  and  implement  initiatives  to  address  survey 
results 

The  survey  findings  may  provide  feedback  that  requires  education  providers  to  develop  or 
enhance  initiatives  to  address  new  priorities,  enhance  services,  meet  expectations,  etc.  The 
results  may  identify  what  works  well  and  what  may  need  to  modified  or  started. 

The  survey  results  may  also  have  direct  relevance  and  influence  to  business  or  education 
plan  developed  by  school  authorities  and  schools. 

Gain  additional  feedback 

Several  school  authorities  use  Parent  Councils  to  from  stakeholders 

explore  issues  drawn  out  the  survey  in  greater 

detail  In  some  cases,  the  findings  of  satisfaction  surveys 

may  need  further  examination.  Additional 
research  may  be  needed  to  clarify  information. 

There  are  various  research  techniques  that  may  be  undertaken  such  as  focus  groups,  in- 
depth  interviews,  and  additional  surveys. 

Organizations  may  consider  further  research  opportunities  with  targeted  populations  in  cases 
where  new  issues  are  identified  from  the  survey  data. 

Evaluate  the  succ^esses  and  challenges  of  the  survey 
projecrt 

Researchers  who  conduct  satisfaction  surveys  are  encouraged  to  assess  the  effectiveness  and 
value  of  the  research  project  and  provide  suggestions  for  improving  the  project  or,  if 
applicable,  recommend  the  need  for  additional  research  or  waves  of  the  survey. 

The  survey  project  and  its  results  may  be  evaluated  formally  or  informally.  In  other  words, 
sometimes  the  evaluation  will  be  thorough,  structured  and  formal,  while  other  times  it  will 
be  impressionistic,  subjective  and  informal.  The  choice  of  process  will  depend  on  resources 
and  interests  of  the  organizadon  that  directs  the  survey  research. 
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The  basic  framework  for  a  project  evaluation  is  to  determine  what  went  well,  and  what  did 
not.  Researchers  might  examine  the  survey  management,  methodologies  or  results  to  identify 
strengths  and  weaknesses. 

Solicit  opinions  from  decision  makers,  stakeholders  and  partners  in  the  evaluation.  Request 
feedback  on  how  survey  design  and  administration  can  be  improved. 

Identify  areas  for  ftfture  satisfaction  measurement 

The  evaluation  may  provide  conclusions  or  assess  areas  for  improvement  for  future 
satisfaction  survey  projects.  The  following  approaches  highlight  the  results  that  might  be 
provided  from  the  project  evaluation: 

•  Assess  if  and  how  the  results  were  actually  used  for  decision  making. 

•  Render  a  judgment  on  the  value  of  the  survey  project. 

•  Assist  decision  makers  in  determining  whether  to  conduct  the  survey  research  project 
again  in  the  future. 

•  Identify  areas  for  improvement. 
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Other  Issues  Related  to  Conducting 
Satisfiaction  Surveys 

Surveying  Children  and  Youth 

special  care,  attention  and  precautions  should  be  taken  when  conducting  survey  research 
with  children  and  youth. 

School  authorities  and  schools  within  Alberta's  education  system  have  policies  and 
procedures  (formal  or  informal)  related  to  survey  research  projects  involving  their  students. 
Schools  or  school  boards  who  are  planning  to  survey  children  or  youth  in  Alberta's 
education  system  should  gain  proper  authorization  within  the  policies  and  procedures  of 
their  jurisdictions.  Organizations  who  are  conducting  surveys  with  students  are  encouraged 
to  contact  policy  administrators  early  in  the  survey  planning  process  to  determine  the 
requirements  and  specifications  needed  to  proceed  with  a  satisfaction  survey. 

Alberta  Education  has  established  the  following  guidelines  for  conducting  survey  research 
with  students  within  the  education  system  (Survey  Research  Policy,  Guidelines  and  Best 
Practices,  2002): 

XX^en  a  child  in  elementary'  school  is  a  respondent,  parents  or  guardians  must  be  informed 
of  the  survey  research  project  before  conducting  an  interview  (interviewer-administered  or 
self-administered  interviews).  - 


-  Section  34  (2)  of  the  Freedom  of  Information  and  Protection  of  Privacy  Act  imposes  a  dut}'  on  public  bodies 
(government  departments,  agencies,  boards,  commissions,  etc.)  to  ensure  that  individuals  are  properly  informed 
about  the  purposes  for  which  their  personal  information  will  be  used.  In  1999,  Alberta's  Information  and 
Privacy  Commissioner  proposed  that  eiementar)'  school  children  may  not  fully  comprehend  the  purposes  for 
which  their  personal  information  will  be  used  or  raise  critical  questions  about  providing  personal  situations.  As 
such,  the  Commissioner  recommended  that  "in  situations  involving  the  collection  of  personal  information 
from  elementary  school  children,  it  is  recommended  the  Public  Body  inform  parents/guardians,  in  writing  of 
the  following: 

•  The  purpose(s)  for  which  the  information  will  be  used. 

•  The  legal  authority'  for  the  collection. 

•  The  titie  address  and  phone  number  of  a  person  who  can  answer  questions. 

•  How  the  child's  information  will  be  used. 

•  The  organizations  involved  in  the  study. 

•  The  measures  in  place  to  ensure  protection  of  personal  information,  and 

•  That  participation  in  the  survey  is  voluntary." 
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When  a  youth  in  a  junior  or  senior  high  is  a  respondent,  a  parent,  guardian  or  relevant  adult 
(such  as  a  teacher)  must  be  informed  of  the  project. 

In  any  other  environment  such  as  the  home,  school  yard  or  other  public  places,  the  child's  or 
youth's  parent  or  guardian  must  provide  consent  before  the  child/ youth  is  approached  for 
an  interview. 

Consent  by  a  responsible  adult  should  not  be  interpreted  as  constituting  permission,  as  the 
child/youth  must  be  granted  an  opportunity  to  accept  or  decline  his/her  participation  in  the 
interview.  A  child/ youth  must  not  be  approached,  under  any  circumstances,  unless  the 
child/youth  is  accompanied  by  his/her  parent,  guardian  or  a  relevant  adult  (such  as  a 
teacher). 

When  requesting  permission  to  interview  a  child/ youth,  sufficient  information  must  be 
given  to  the  parent,  guardian  or  a  relevant  adult  (such  as  a  teacher)  for  him/her  to  give 
adequate  consideration  to  the  decision  about  granting  permission  for  the  interview.  The 
t}^es  of  information  to  provide  parents,  guardians  or  a  relevant  adult  are  covered  in  Section 
34(2)  of  the  Freedom  of  Information  and  Protection  of  Privacy  Act. 

It  is  desirable  to  have  the  parent,  guardian  or  a  relevant  adult  (such  as  a  teacher)  close  at 
hand  while  the  interview  is  being  conducted. 

While  it  may  be  imperative  to  avoid  certain  subjects  when  interviewing  children  (e.g.  a  topic 
that  might  frighten  a  child),  a  similar  research  subject  may  be  covered  with  youth  if 
appropriate  precautions  are  taken.  Research  topics  that  may  need  special  care  or  precautions 
include: 

•  Issues  that  could  upset  or  worr^'  the  child/ youth. 

•  Those  that  risk  creating  tension  between  the  child/ youth  and  his/her  parents. 

•  Those  relating  to  potentially  sensitive  family  situations  (e.g.  parental  relationships,  income, 
use  of  stimulants,  and  family  illnesses). 

•  Those  relating  to  race,  religion,  or  similar  socially  or  politically  sensitive  matters. 

•  Those  concerned  with  sexual  activities. 

•  Those  related  to  illegal  or  socially  unacceptable  activities. 

The  overall  welfare  and  well-being  of  the  child/ youth  participating  in  survey  projects  should 
be  given  utmost  consideration.  In  all  cases,  the  safety,  rights  and  interests  of  the  child  or 
youth  must  be  advocated  and  upheld.  All  survey  research  carried  out  with  children  or  youth 
must  be  conducted  to  high  ethical  standards  so  that  no  abuse,  real  or  perceived,  is  caused  to 
the  child  or  youth  involved. 
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Freedom  of  Informatioti  and  Protection  of  Privacy 
Specifications 

Satisfaction  surveys  involve  the  collection,  use,  retention,  disclosure  and  disposition  of 
personal  information.  Part  2  of  the  Freedom  of  Information  and  Protection  of  Privacy  Act 
(FOIP  Act)  and  the  FOIP  Regulation  have  established  privacy  protection  provisions  for 
how  organizations  can  gather,  use  or  disclose  personal  information. 

Researchers  should  refer  to  the  following  documents  in  the  planning  phase  of  the 
satisfaction  survey  project  to  ensure  that  the  survey  adheres  to  FOIP  protocols: 

FOIP:  Conducting  Surveys:  A  Guide  to  Privacy  Protection,  Revised  August 

2003.http:/ / www3.gov.ab.ca/ foip/ other_resources/ publications_videos/ survey _guide.cfm 

FOIP:  Contract  Manager's  Guide,  December  2003. 

http://www3.gov.ab.ca/ foip/ other_resources/ publications_videos/contract_managers_guide. 
cfm 

The  Information  Management,  Access  and  Privacy  Branch  of  Alberta  Government  Services 
has  prepared  a  'Best  Practices  Checklist'  to  assist  government  staff  in  addressing  privacy 
compliance  challenges  as  it  relates  to  survey  research.  This  Checklist  is  presented  in 
Appendix  I  for  consideration.  Jurisdictions  who  are  engaged  in  survey  research  are 
encouraged  to  review  the  criteria  presented  in  the  'Checklist'  to  ensure  the  survey  projects 
comply  with  the  FOIP  Act.  Staff  are  also  encouraged  to  consult  with  the  Department's 
FOIP  Coordinator  to  seek  advice  on  privacy  and  compliance  issues. 
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Definition  of  Terms 

The  following  are  definitions  of  terms  commonly  used  in  survey  research,  and  presented  in 
this  document: 

•  Anonymity  -  Information  about  respondents  must  be  reported  in  ways  that  do  not 
permit  the  identification  of  any  individual. 

•  Bias  -  The  tendency,  during  any  step  in  a  survey,  to  systematically  favor  or  give  advantage 
to  answers  or  findings  which  will  cause  resulting  estimates  to  deviate  in  one  direction 
from  the  true  value.  Bias  may  or  may  not  be  intentional. 

•  Census  -  A  survey  in  which  the  researcher  attempts  to  gather  data  from  all  members  of  a 
population. 

•  Coding  -  A  process  of  converting  questionnaire  information  to  numbers  or  symbols  to 
facilitate  subsequent  data  processing  and  analysis. 

•  Codebook  -  A  set  of  question  responses  and  their  associated  values  (code  numbers). 
Typically,  codebooks  are  used  to  document  categories  or  values  assigned  to  question 
responses  on  a  survey  questionnaire. 

•  Computer-assisted  telephone  interviewing  (CAT!)  -  A  type  of  telephone 
interviewing  in  which  interviewers  enter  responses  to  questions  into  a  computer  as  they 
are  received.  A  computer  viewing  screen  automatically  displays  questions  that 
interviewers  ask  of  respondents  through  a  telephone. 

•  Confidentiality  -  The  situation  where  the  privacy  of  information  provided  by  individual 
respondents  to  a  survey  is  maintained  and  the  information  about  individual  respondents 
cannot  be  derived  from  the  published  survey  results. 

•  Consent  -  Respondents  (or  appropriate  parents/guardians)  providing  permission  or 
approval  to  be  interviewed.  Respondents  have  the  right  to  terminate  interviews  at  any 
time  and  to  withhold  any  information  they  choose. 

•  Data  -  Collective  reference  to  individual  items  of  information. 

•  Data  Cleaning  -  The  application  of  procedures  of  coding  and  identifying  missing, 
invalid  or  inconsistent  entries. 

•  Interview  -  Any  form  of  direct  or  indirect  contact  with  respondents  where  the  object  is 
to  acquire  data  or  information  which  could  be  used  in  whole  or  in  part  for  the  purposes 
of  a  survey  research  project. 

•  Open  Competition  -  Service  requirements  are  advertised  to  the  general  public  and  all 
interested  parties  are  invited  to  submit  a  proposal  (e.g.  through  the  MERX  system  or 
through  an  advertisement  in  provincial  newspapers). 

•  Personal  Information  -  Relates  to  data  or  information  about  an  identifiable  individual. 
It  includes  an  individual's  name,  address,  telephone  number,  age,  gender,  marital  stams, 
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educational  and  employment  history,  and  personal  opinions  and  views. 3  A  complete 
definition  is  available  in  the  FOIP  Act,  section  1  (n). 

•  Pilot  Test  -  A  small  scale  survey,  using  respondents  from  the  target  population  for  the 
purpose  of  testing  the  integrated  functioning  of  all  component  parts  of  the  survey 
operation.  Revisions  can  then  be  made  as  necessary  before  the  full-scale  survey  is 
undertaken. 

•  Precision  ~  A  measure  of  the  closeness  of  the  sample  estimates  to  the  result  from  a 
census  taken  under  identical  conditions. 

•  Pre-Test  -  A  preliminary  testing  of  individual  component  parts  of  a  survey  to  check  that 
each  component  fianctions  as  planned.  Each  component  can  then  be  revised  as  needed. 

•  Probability  Sample  -  Any  method  of  selection  of  units  from  a  population  in  which: 

°     every  unit  has  a  known  and  calculable  chance  (greater  than  zero)  of  being 
selected, 

°     the  sample  is  drawn  by  a  random  selection  procedure,  and 
°     the  probabilities  of  selection  are  used  to  project  the  results  from  the  sample  to 
the  population. 

•  Questionnaire  -  A  series  of  structured  questions  designed  to  elicit  information  on  one 
or  more  topics  from  a  respondent. 

•  Rating  scale  -  A  type  of  survey  question  designed  to  record  direction  and  strength  of  a 
respondent's  perception  toward  a  specific  topic. 

•  Reliability  -  The  extent  to  which  a  survey,  if  repeated  using  another  (but  statistically 
equivalent)  sample  and  identical  questionnaire  and  procedures,  would  produce  the  same 
results. 

•  Representative  sample  surveys  -  Surveys  in  which  the  sample  is  a  selection  from  a 
larger  population  having  the  essential  characteristics  of  the  total  population. 

•  Request  for  Proposals  (RFP)  —  A  document  specifying  the  requirements  of  the 
survey  project  that  is  sent  out  to  contractors.  The  contractors  then  reply  (if  interested) 
with  proposals  based  on  these  requirements. 

•  Respondent  -  Any  individual  or  organization  from  whom  any  information  is  sought  for 
the  purposes  of  a  survey  research  project.  The  term  covers  cases  where  information  is  to 
be  obtained  by  verbal  interviewing  techniques,  postal  and  other  self-completion 
questionnaires,  mechanical  or  electronic  equipment,  observation  and  any  other  method 
where  the  identity  of  the  provider  of  information  may  be  recorded  or  otherwise  traceable. 


Freedom  of  Information  and  Protection  of  Privacy,  Conducdng  Surv^eys:  A  Guide  to  Privacy  and  Protection,  ISBN  0- 
7785-1799-3,  Government  of  Alberta,  September  2001. 
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•  Rotations  -  The  manner  in  which  various  questions  or  items  within  a  question  are  asked 
or  shown  in  different  order  for  each  interview.  This  process  helps  to  eliminate  order  bias 
that  might  develop  if  questions  were  asked  or  shown  in  exacdy  the  same  order  for  each 
interview  conducted  in  a  particular  survey. 

•  Sample  frame  -  Any  list,  material,  or  device  that  identifies,  and  allows  access  to 
members  of  a  target  population. 

•  Screening  -  The  process  of  checking  whether  an  individual  or  a  situation  should  be 
included  in  a  survey  or  survey  question. 

•  Selected  Competition  -  A  process  where  specified  contractors  are  invited  to  respond 
to  a  Request  for  Proposal. 

•  Single  Source  Selection  -  A  process  where  one  contractor  is  iavited  to  propose  on  the 
provision  of  services. 

•  Skips  -  A  device  used  in  questionnaires  to  guide  respondents  (or  interviewers)  past  a  (set 
of)  question(s)  that  do  not  apply  to  a  particular  respondent. 

•  Survey  -  The  term  is  used  to  refer  to  the  general  method  of  data  gathering,  wherein  a 
number  of  respondents  are  asked  identical  questions  through  a  systematic  questionnaire 
or  interview.  Instruments  used  to  collect  actual  data  will  be  referred  to  as  'questionnaire 
(s)'  or  'survey  instrument  (s).' 

•  Survey  Instrument  -  Any  device  used  to  solicit  or  gather  data  or  information  from  a 
respondent,  for  example,  introduction  letters,  questionnaires,  computers,  tape  recorders, 
or  video  tape  machines. 

•  Target  Population  -  The  complete  group  of  units  to  which  survey  results  are  to  apply. 
These  units  may  be  persons,  animals,  objects,  businesses,  trips,  etc. 

•  Validity  -  The  degree  to  which  a  method  of  measurement  succeeds  in  measuring  what  it 
is  intended  to  measure. 

•  Wave  -  A  set  of  activities  in  a  survey  or  series  of  surveys  conducted  among  the  same 
population  using  the  same  methodology. 
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APPENDIX  A  -  Templates  for  Evaluation  of  Survey 
Consultants 

This  form  provides  a  framework  for  evaluating  proposals  submitted  by  outside  contractors  for  survey  research. 

The  evaluation  process  involves  assigning  rating  scores  (1  being  'does  not  meet  requirements'  and  5  being  'exceeds 
requirements')  to  various  criteria  depending  on  how  well  contractors  proposals  meet  the  requirements  of  each  criterion. 
In  some  cases,  there  may  be  some  qualifying  criteria  required  for  a  proposal  to  be  considered.  A  score  of  1  (does  not 
meet  requirements)  or  2  (somewhat  below  requirements)  would  cause  the  immediate  disqualification  of  a  proposal.  In 
order  to  expedite  what  can  be  a  lengthy  process,  these  criteria  may  be  evaluated  in  isolation.  Note  that  the  form  begins 
with  space  for  these  criteria. 

Each  criterion  can  be  weighted  to  reflect  the  relative  importance  of  the  requirements  to  the  survey  research  project  (e.g. 
one  criterion  may  be  more  important  to  the  project  than  another).  Assign  a  weight  to  each  criterion  (below).  The  sum 
of  all  the  weights  should  equal  20.  Then  record  the  weight  for  each  category  in  the  space  provided  in  the  table,  and 
assign  a  score  for  each  criterion.  You  may  want  to  include  specific  comments  (strengths  and  weaknesses)  regarding  the 
scores  given. 


After  scoring  each  category,  multiply  the  scores  by  the  assigned  weights.  Add  the  weighted  scores  and  record  the 
total  in  the  box  for  the  Overall  Weighted  Score. 

When  scoring,  use  the  following  scale:  Assign  weights  to  categories:  Weights: 

1  -  Does  not  meet  requirements 

2  -  Somewhat  below  requirements 

3  —  Adequately  meets  requirements 

4  -  Meets  requirements  very  well 

5  —  Exceeds  requirements 


Assign  weights  to  categories: 

Understanding  of  Project  &  Requirements 
Approach/Methodology 
Project  Timelines  &  Cost 
Experience  &  Stature 
Ability  to  Complete  the  Project 
Overall  Assessment 


Mandatory 
Criteria 

Comments 

Evaluation 

Strengths 

Weaknesses 

Score 

(1  or  2  causes  immediate 
disqualification) 

Criteria 

Comments 

Evaluation 

Strengths 

Weaknesses 

Score    X      Weight  =  Weighted 
°  Score 

Understanding  of 
Project  & 
Requirements 

Approach/ 
Methodology 

Project  Timelines 
&  Costs 

Experience  & 
Stature 

Abilit)'  to 
Complete  the 
Project 

Overall 
Assessment 

Additional 
Criterion 

Overall  Weighted  Score: 

51 


A  Practical  Guide  to  Conducting  Surveys  wit/iin  Alberta K-12  Education  System 


The  following  summaries  present  infomiation  that  can  be  used  to  evaluate  survey  research 
proposals  submitted  by  outside  contractors: 

/.         Understanding  of  Project  (& 
Requirements 

•  General  understanding 

°     Is  the  methodology  appropriate  for  the  project  requirements? 
°     Does  the  proposal  demonstrate  understanding  of  the  essential  aspects  of  the 
project? 

°     Will  the  end  report  address  the  objectives  of  the  project  based  on  the  proposal? 

•  Detailed  understanding 

°     Does  the  proposal  demonstrate  understanding  of  the  issues  involved? 
°     Can  the  firm  anticipate  potential  problems? 
°     What  solutions  does  the  firm  propose? 

2.  Approach /Methodologji 

•  Methodological  reliability 

°     Does  the  proposal  provide  a  high  probability  of  accomplishing  the  study 
objectives? 

•  Quality  of  data 

°     Is  the  accuracy  of  the  data  ensured? 

•  Statistical  precision 

°     Is  the  reliability  of  the  data  ensured? 

•  Technical  integrity 

°     Is  the  Approach /Methodology  based  on  sound  research  design  principles? 

•  Flexibility  of  design 

°     Is  there  enough  flexibility  to  ensure  success  if  project  parameters  change? 

3.  Project  Timelines  (&  Costs 

•  Budget 

°     How  does  the  cost  compare  to  the  budget?  Is  it  reasonable? 
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Have  the  components  of  the  budget  been  explained  clearly? 


•  Timelines 


Are  the  proposed  timelines  agreeable  with  the  project  requirements? 
Have  the  components  of  the  timeline  been  explained  clearly? 


4.  Experience  <&  Stature 

•  Education/Technical  expertise 

°     How  much  overall  experience  does  the  personnel/ firm  have  in  data  collection 

and  data  analysis/ report  writing? 
°     Does  the  firm  possess  sufficient  background  knowledge  for  the  project? 

•  Experience 

°     Does  the  firm  have  experience  in  similar  and/ or  related  projects  (documented  in 
project  Hsts)? 

•  Credibility 

°     Strong  references  from  previous  clients? 

5.  Ability  to  Conjplete  the  Project 

•  Resources 

°     Does  the  firm  have/have  access  to  sufficient  resources  to  complete  the  project? 
°     Who  are  the  key  personnel  to  be  involved  in  the  project,  and  how  qualified  are 
they? 

°     How  involved  are  key  personnel  in  each  phase? 

•  Management/Supervisory  resources 

°     Does  the  firm  demonstrate  an  ability  to  manage  the  various  aspects  of  the 
project,  and  have  they  allocated  sufficient/ appropriate  resources  for  project 
management  and  supervision? 


Overall  Assessment 
•  Overall  quality 

°     Was  adequate  care,  attention  to  detail  and  effort  put  into  planning/ creation  of 
the  proposal? 
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•  Style 

°     Clear/ concise  writing;  carefully  edited;  well  presented  concepts;  attractive  and 
easily  understood  layout? 

•  Involvement 

°     Have  provisions  been  identified  to  keep  the  clients  involved?  How/how  much? 

•  Strategy 

°     Does  the  proposal  provide  a  superior  strategy  for  accomplishing  goals  of  the 
project? 

°     Does  the  firm  add  extra  value  to  the  project? 

7.        Additional  Criterion 

•  Issues  or  specifications  customized  to  the  project. 
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Project  Name:  Survey  of  Subject  Areas  of  Interest  to  High  School  Students 

Name  of  Company/Firm:  ABC  Research  and  Consulting 

Name  of  Reviewer:  Jane  Smith  Date  of  Review:  July  21,  2005 


This  form  provides  a  framework  for  evaluating  proposals  submitted  by  outside  contractors  for  survey  research. 

The  evaluation  process  involves  assigning  rating  scores  (1  being  'does  not  meet  requirements'  and  5  being  'exceeds 
requirements')  to  various  criteria  depending  on  how  well  contractors  proposals  meet  the  requirements  of  each  criterion.  In  some 
cases,  there  may  be  some  qualifying  criteria  required  for  a  proposal  to  be  considered.  A  score  of  1  (does  not  meet  requirements) 
or  2  (somewhat  below  requirements)  would  cause  the  immediate  disquaUfication  of  a  proposal.  In  order  to  expedite  what  can  be 
a  lengthy  process,  these  criteria  may  be  evaluated  in  isolation.  Note  that  the  form  begins  with  space  for  these  criteria. 

Each  criterion  can  be  weighted  to  reflect  the  relative  importance  of  the  requirements  to  the  survey  research  project  (e.g.  one 
criterion  may  be  more  miportant  to  the  project  than  another).  Assign  a  weight  to  each  criterion  (below).  The  sum  of  all  the 
weights  should  equal  20.  Then  record  the  weight  for  each  category  in  the  space  provided  in  the  table,  and  assign  a  score  for 
each  criterion.  You  may  want  to  include  specific  comments  (strengths  and  weaknesses)  regarding  the  scores  given. 

After  scoring  each  category,  multiply  the  scores  by  the  assigned  weights.  Add  the  weighted  scores  and  record  the 
total  in  the  box  for  the  Overall  Weighted  Score. 


When  scoring,use  the  following  scale: 

1  -  Does  not  meet  requirements 

2  —  Somewhat  below  requirements 

3  -  Adequately  meets  requirements 

4  -  Meets  requirements  very  well 

5  —  Exceeds  requirements 


Assign  weights  to  categories:  Weights: 

Understanding  of  Project  &  Requirements   5_ 

Approach/Methodology   3_ 

Project  Timelines  &  Cost   4_ 

Experience  &  Stature   2_ 

Ability  to  Complete  the  Project  4 
Overall  Assessment  2 


Mandatory  Criteria 

Comments 

Evaluation 

Strengths 

Weaknesses 

Score 

(1  or  2  causes  immediate 
disqualification) 

Bid  is  within  Budget 

Budget  =  $12,000 
Bid  =  $11,275 

5 

Criteria 

Comments 

Evaluation 

Strengths 

Weaknesses 

Score    X      Weight  =  Scorc^^^^ 

Understanding  of 
Project  & 
Requirements 

Telephone  survey 
proposed,  matches 
objective  of  validit}' 

Limited  depth  of 
interviewing  in 
telephone  survey 

3 

5 

15 

Approach/ 
Methodology 

Quality  control 
measures  documented 

Non-standard  qualit}' 
control  measures 

4 

3 

12 

Project  Timelines  & 
Costs 

Well  within  budget 
boundary 

TimeUnes  are  not 
detailed,  may  be  some 
risk  of  late  delivery  of 
results 

3 

4 

12 

Experience  & 
Stature 

Firm  is  well-known  in 
this  market 

Firm  has  not  completed 
comparable  project 

3 

2 

8 

Ability  to  Complete 
the  Project 

Firm  has  large 
professional  team 

Firm  has  larger  client 
that  may  distract 
resources  during  our 
project 

2 

4 

8 

Overall  Assessment 

Firm  principals  present 
well,  understand 
project 

Some  concerns  about 
importance  of  other 
chents 

3 

2 

6 

Additional  Criterion 

None 

Overall  Weighted  Score: 

61 
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APPENDIX  B  -  Scale  Response  Questions 

The  following  four  step  process  is  offered  to  researchers  to  assist  in  the  development  of 
response  scale  questions: 

STEP  1:  Develop  a  distinct  list  of  items  to  be 
measured. 

•  Determine  and  clarify  the  concepts,  ideas  or  theories  to  be  measured. 

•  Determine  the  level  of  specificity  or  generality  required  from  the  results  to  address  the 
business  need  being  addressed. 

•  Determine  whether  items  of  concepts,  ideas  or  theories  being  measured  are  distinct 
(otherwise  respondents  may  experience  challenges  in  responding  to  the  questions). 

•  The  items  in  the  list  should  not  be  ambiguous. 

•  Avoid  lengthy  wording  that  compromises  clarity  (though  not  at  the  expense  of  meaning). 

•  Avoid  multiple  negatives. 

•  Avoid  double-barreled  items  where  the  items  convey  2  or  more  ideas. 

•  Avoid  ambiguous  pronoun  references. 

•  Use  both  positively  and  negatively  worded  items  to  avoid  bias  where  respondents  are 
inclined  to  affirm  items  regardless  of  their  content  (however,  there  may  be  a  trade-off 
between  avoiding  bias  and  creating  confusion). 

STEP  2:  Determine  the  measurement  format. 

Measurement  formats  (e.g.  satisfaction,  agreement,  interested,  etc.)  should  be  compatible 
with  the  items  generated  for  the  question. 

The  following  presents  aspects  of  scale  portions  that  researchers  should  consider  when 
deciding  upon  an  appropriate  measurement  format: 

Number  of  categories  -  The  number  of  categories  or  options  selected  by  the  researcher 
should  provide  sufficient  discrimination  within  the  scale.  In  other  words,  the  higher  the 
number  of  categories  or  options,  the  finer  the  distinctions  available  for  analysis.  However, 
shorter  scales  have  the  advantage  that  they  are  less  of  a  burden  on  respondents.  It  is 
important  for  researchers  to  consider  whether  respondents  will  find  the  distinctions  relevant 
and  meaningful  (e.g.  can  the  respondent  actually  distinguish  a  difference  between  somewhat 
agree  and  slightiy  agree?). 
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4-Point  Scale  7-Point  Scale 

Strongly  agree  Strongly  agree 

Agree  Somewhat  agree 

Disagree  Slightly  agree 

Strongly  disagree  Neither  agree  nor  disagree 

SHghdy  disagree 
Somewhat  disagree 
Strongly  disagree 


Balanced  or  unbalanced  scales  -  The  researcher  must  decide  whether  the  scale  should  be 
balanced  or  unbalanced.  The  balanced  scale  provides  an  equal  number  of  response 
categories  on  both  ends  of  the  continuum.  This  is  the  most  common  form  of  rating  and 
generally  satisfies  the  requirements  for  interval  measurement.  Conversely,  an  unbalanced 
scale  is  used  when  the  direction  of  response  is  generally  known  and  finer  distinctions  on  one 
end  of  the  continuum  are  desired.  Although  unbalanced  scales  are  sometimes  used  in  social 
and  marketing  research,  they  are  generally  not  recommended  because  the  question  has  the 
potential  to  bias  the  respondent  and  it  may  be  difficult  to  analyze  and  interpret  the  data. 


Balanced  Scale  Unbalanced  Scale 

Strongly  agree  Very  strongly  agree 

Somewhat  agree  Strongly  agree 

Neither  agree  nor  disagree  Somewhat  agree 

Somewhat  disagree  Somewhat  disagree 

Strongly  disagree  Disagree 


Odd  or  Even  Number  of  Categories  -  When  balanced  scales  are  used,  researchers  must 
determine  whether  the  scale  will  have  an  odd  or  even  number  of  categories  (the  use  of  odd 
number  of  categories  typically  designates  the  middle  category  as  'neutral'  such  as  neither 
agree  nor  disagree,  except  in  the  application  of  an  unbalanced  scale).  The  decision  to  use 
either  odd  or  even  number  of  response  categories  usually  depends  on  the  researcher's 
assumptions  about  respondents'  mindset.  Researchers  who  advocate  even-numbered 
categories  generally  propose  that  respondents  may  use  the  neutral  options  to  hide  their 
opinion  and  they  should  be  forced  to  indicate  some  degree  of  opinion.  Advocates  of  odd- 
numbered  categories  suggest  that  respondents  can  be  neutral  in  their  opinion  or  perspectives 
and,  thus,  should  be  allowed  to  state  their  ambivalence. 


4-Point  Scale  5-Point  Scale 

Strongly  agree  Strongly  agree 

Agree  Somewhat  agree 

Disagree  Neither  agree  nor  disagree 

Strongly  disagree  Somewhat  disagree 

Disagree 


The  following  are  general  considerations  for  using  odd  or  even  number  of  categories: 
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Even  Number  of  Categories  (such  as  a  4  point  scale)  -  Is  appropriate  in  situations  where 
a  'forced'  choice  is  necessary.  However,  it  should  be  noted  that  forced-choice  may  increase 
non-response  rate. 

Odd  Number  of  Categories  (such  as  a  5  point  scale)  -  Is  appropriate  if  a  neutral  opinion 
is  an  appropriate  response.  But,  the  researcher  may  have  difficulties  interpreting  the  neutral 
proportion  of  responses. 

STEP  3:  Have  question  items  and  response 
scales  reviewed  by  survey  committee  members 
who  have  survey  expertise. 

Issues  that  should  be  considered  when  questions  and  response  scales  are  being  reviewed, 
include: 

•  Relevance  of  each  item  and  what  it  is  intended  to  measure. 

•  Clarity  and  conciseness  of  items. 

•  The  potential  for  response  bias. 

STEP  4:  Evaluate  response  scale  questions  in  a 
survey  pre-test. 

De-brief  interviewers  or  pilot-test  respondents  to  determine  whether  items  are  clear, 
understandable,  and  distinct  in  meaning. 

Review  the  performance  of  items  to  assess  variances  in  responses  (high  variance  of  scale 
items  is  desirable). 

If  there  are  concerns  about  a  response  scale  question,  try  alternatives  in  the  pre-test. 
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APPENDIX  C  -  Call  Record  Management  Form 


Interviewer  # 

Date: 

Total 

(Project  Name) 

B  -  Busy 

Final 

Complete 

NA  -  No  Answer 

X   IIWX-IV^  IT 

Call  1 

Call  2 

Call  3 

Status 

Number 

AM  -  Answering 

ccc  1  c-ic 
DDD- i  DJD 

IVTiirViinp 

i.VXclV'X  1  f  i  x^ 

QQ1  9 

CB  -  Arranged 

Call  Back 

555-8175 

555-2978 

C  -  Completed 

555-8304 

R  -  Refused 

555-5678 

Fax  -  Fax 

555-3497 

I  -  Incomplete 

555-0987 

T  -  Terminated 

555-6821 

NQ  -  Not 
Qualified 

555-4093 

NIS-Notin 
Service 

555-0112 

Bus  -  Business  # 

555-7891 

L-Language 

555-8374 

Total  CaUs  made 

555-5930 

555-2947 

NOTES: 

555-4930 

555-0081 

555-4581 

555-7623 

555-6611 

555-9641 

555-1398 

555-7549 

555-4760 

555-3322 

555-8861 

555-2875 

555-6830 

555-1881 
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APPENDIX  D  -  FOIP  Best  Practices  Checklist 


(Source:  Freedom  of  Information  and  Protection  of  Privacy:  Conducting  Surveys:  A  Guide  to  Privacy 
Protection,  ISBN  0-7785-2097-8,  Government  of  Alberta,  revised  August  2003) 

□  Clearly  define  the  issues  you  wish  to  address 
Planning  and  Design  through  survey  research.  This  wiU  help  limit 

collection  of  information  to  that  which  is 
necessary  to  address  the  issues. 

□  If  the  survey  is  going  to  gather  large  amounts  of  personal  information,  very  sensidve 
personal  information  and/or  retain  personal  information  for  a  length  of  time,  a  privacy 
impact  assessment  should  be  performed. 

□  Ensure  staff  has  a  clear  understanding  of  the  privacy  issues  before  beginning  survey 
research. 

□  If  using  an  external  contractor  for  any  stage  of  the  project,  have  a  written  agreement  or 
contract  in  place  ensuring  compliance  with  the  FOIP  Act. 

□  Whenever  possible,  design  the  survey  so  that  no  personal  information  is  collected. 

□  If  the  survey  cannot  be  carried  out  anonymously,  design  it  so  that  personal  information 
is  transformed  before  use  or  disclosure. 

□  If  using  coded  surveys,  ensure  that  procedures  are  in  place  to  minimize  the  extent  of 
access  to  both  sets  of  data. 

□  Make  sure  the  survey  participants  are  informed  of  the  purpose  of  the  survey  and  how 
you  will  be  using  any  personal  information  that  may  be  transformed. 

□  When  you  know  in  advance  that  client 
Sampk  Selection  information  will  be  used  to  select  a  survey 

sample,  provide  notice  of  this  use  at  the  time 
of  collection. 

□  When  you  have  not  anticipated  use  of  personal  information  to  select  a  survey  sample  at 
the  time  of  collection,  use  that  information  only  if  the  use  is  consistent  with  the  original 
purpose  of  collection  or  you  have  individual  written  consent. 

□  If  you  are  asking  the  Information  and  Privacy  Commissioner  for  permission  to  collect 
personal  information,  complete  a  privacy  impact  assessment  to  demonstrate  the  need. 

□  Before  sharing  data  to  select  a  survey  sample  make  sure  there  is  authority  to  collect  and 
disclose,  and  a  personal  information  sharing  agreement  is  in  place. 

□  If  possible,  avoid  indirect  collection  of  personal  information  to  obtain  a  survey  research 
sample.  Instead,  have  the  public  body,  or  other  institution  that  maintains  the  personal 
information,  contact  potential  participants  directiy  on  your  behalf 

□  Before  sharing  data  or  contacting  potential  research  participants  on  behalf  of  another 
public  body,  you  should  ensure  that  you  have  the  authority  to  use  or  disclose  the 
personal  information  for  these  purposes  under  section  39  or  40(1)  of  the  FOIP  Act. 

□  When  contacting  potential  research  participants  on  behalf  of  another  public  body,  ensure 
that  replies  go  directiy  to  the  public  body  conducting  the  survey. 

□  Any  collection  of  personal  information  done  on  behalf  of  a  public  body  requires  a  notice 
of  collection. 
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□  If  personal  information  is  collected  for  a 
Data  Collection                                                  purpose  not  directly  related  to  the  survey, 

keep  the  two  types  of  information  separate 
and  the  use  and  disclosure  of  this  information 

should  be  made  clear. 

□  Ensure  you  have  the  authority  to  collect  the  personal  information  required  for  the  survey 
under  section  33  of  the  FOIP  Act. 

□  Before  collecting  personal  information  indirectly  ensure  you  have  the  authority  to  do  so 
under  section  34(1)  or  (3)  of  the  FOIP  Act. 

□  Limit  the  amount  of  personal  information  collected  to  what  is  strictly  necessary. 

□  When  contacting  potential  survey  participants,  take  steps  to  protect  their  privacy  by  not 
disclosing  to  third  parties  the  name  of  your  institution  or  the  reason  for  contacting  the 
potential  survey  participants. 

□  Unless  the  survey  is  done  anonymously,  provide  assurances  of  confidentiality  only  with 
the  proviso  that  disclosure  of  personal  information  may  occur  if  required  by  statute  or 
the  courts. 

□  When  collecting  personal  information  to  conduct  a  survey,  provide  notice  of  collection 
in  compliance  with  section  34(2)  of  the  FOIP  Act. 

□  Provide  survey  participants  with  sufficient  information  about  the  survey  so  that  they 
understand  the  use  being  made  of  their  personal  information. 

□  Whenever  possible,  collect  personal  information  directiy  from  the  subject  individual. 

□  Obtain  prior  written  consent  from  each  individual  if  the  intent  is  to  disclose  personal 
information  that  could  identify  him  or  her  to  those  the  survey  is  about. 

□  Use  and  disclose  personal  information  only 
Data  Analysis  for  the  purposes  specified  to  the  survey 

participants  at  the  time  of  collection. 

□  Before  using  personal  information  for  a 
purpose  not  specified  at  the  time  of  collection,  obtain  the  individual's  written  consent. 

□  Report  survey  results  as  aggregate 
porting  Results  information. 

□  Do  not  report  results  of  small  cells  (i.e.  5  or 
fewer  participants). 

□  Consider  other  ways  of  transforming  personal  information  into  non-identifiable 
information. 

□  Whenever  possible  store  personal 
Records  Management  information  separately  from  the  survey 

responses. 

□  Keep  a  record  of  the  fact  that  a  personal 
information  bank  is  used  to  select  survey  samples. 

□  Ensure  that  you  have  a  records  retention  and  disposition  schedule  in  place  for  all  records 
related  to  the  survey  and  follow  it. 
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APPENDIX  E  -  Error  Checks  for  Satisfaction  Surveys 


Error  Type 

Problem 

How  Error  may  be 
Addressed 

Sampling 
errors 

Target  population 
inadequately  defined 

Invalid  survey  results 

Re-defmed  survey 
population 

Poor  sample  frame 

Potential  respondents  will 
be  missed  or 
inappropriate 
respondents  included 

Clean  and  update 
sample  frame 

Data 

collection 

errors 

Interviewer  errors: 

-  failing  to  ask 
questions  properly 

-  not  followinp; 
questionnaire  flow 
-leading  respondents 

-  improper  recording 
of  answers 

Inaccurate  or  poor  data 

-  proper  selection, 
training  and 
supervision  of 
interviewers 

Respondent 
errors 

Respondent  provide 
incorrect  information 

Invalid  data 

-  ensure  questions  are 
worded  and  formatted 
properly 

Respondent  lacks 
required  information 

Question  omitted  or 
guessed  at 

-  allow  for  uncertainty 
in  question  answer 
(don't  know) 

-  ensure  proper 
sample  frame  is  used 

Data 

administration 
errors 

Coding  errors 

Invalid  and  unreliable 
data 

-  pre-code 
questionnaires 

-  check  coding 

-  train  coders 

Data  entry  errors 

Invalid  and  unreliable 
data 

-  use  computer 
software  programs 
customized  for  data 
capture 

-Train  data  entry  staff 

-  Conduct  verification 
of  data  entry 
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